A lot of LLM projects still die in the notebook. Someone wires up a model in a Colab, pastes a key, adds a clever prompt, and gets a compelling demo. Screenshots circulate. Leadership is impressed. The same team then spends twelve months trying to turn that screenshot into something that survives real on the open web traffic, compliance reviews, and incident playbooks outages harms pr crises reports. Most of what people call "MLOps" was built around supervised models with well-defined inputs, outputs, and metrics. LLM systems)-reliability engineering are messier: they are interactive, stateful at the product level, and tied to data and tools that live outside the model. The old patterns are still useful, but they are not enough. The gap between notebook and production is wider than it looks, and it has a specific shape. Once you see the patterns, you can stop improvising architecture from scratch every time someone types a clever prompt. ## The false comfort of the notebook In the notebook, everything is fixed and friendly. There is one model, one prompt, one input. You run a cell, read the output, tweak. There is no concurrency, no rate limits, no retries, no partial failures. You do not think about latency, only about "does this look smart". You also cheat on evaluation. You try five or ten examples, keep the ones that look good, and talk yourself into believing they are representative. You are both the designer and the judge, so every small improvement feels like progress. The moment you leave the notebook you acquire problems that were invisible five minutes earlier: - Inputs become arbitrary user text, not your own carefully chosen examples.

• You have to serve many requests at once and respect SLAs.
• You have to log, debug, and roll back changes.
• You have to answer questions like "why did we give this answer, yesterday, to this user, and what changed since?" None of that lives in a Colab. It lives in architecture and process. ## What is different about MLOps for LLMs Classical supervised models usually have clear structure. You have features in, labels out, a scalar loss, and a small set of metrics you can monitor. You can retrain periodically on batches of labeled data, roll out a new model, and compare curves. LLM-based systems are more like ephemeral programs: - Inputs are free-form instructions and documents.
• Outputs are long-form text, sometimes structured, sometimes not.
• The model can call tools and retrieve data, not just emit tokens.
• Failure modes are qualitative and contextual, not just "the accuracy dropped two points." As a result, production LLM systems lean on a few recurring patterns that sit on top of the old MLOps toolbox. If those patterns are missing, you are effectively still in notebook territory, just with more users. ## Pattern 1: wrap the model in a contract, not just an API The first step is always a wrapper: a service that talks to the model, adds auth, and handles basic retries. Many teams stop there. That is a mistake. You want a contract for what this model-backed endpoint does. Not just input and output types, but behavioral commitments: - What kind of tasks is this endpoint allowed to handle
• What tools or data it is allowed to use
• What invariants must hold on outputs (format, length, fields, prohibited content) That contract is what you can test and monitor. It is also what lets you swap the underlying model later, without breaking every consumer. In practical terms, this means defining internal types and schemas. If you let "prompt in, free-form text out" leak throughout your codebase, you will never claw your way back to sanity. The model is a component. The contract lives above it. ## Pattern 2: make retrieval and tools first-class parts of the system Almost every serious LLM application ends up using retrieval, tools, or both. In production, those are not embellishments; they are core architecture. The notebook view is "I call the model with some context and maybe a function signature". The production view is: - A retrieval layer with its own data pipelines media pipelines from text prompt to production asset, indexes, filters, and evaluations.
• A tool layer with explicit APIs, schemas, error handling, and permissions.
• An orchestration layer that decides when to retrieve, when to call tools, and how to combine results. From an MLOps standpoint, that means: - Versioning indexes and retrieval configurations, not just model weights.
• Monitoring tool call rates, failures, and latencies as carefully as model tokens.
• Treating orchestration logic as code that must be tested, reviewed, and rolled out like any other service. If your entire system logic lives inside prompts you type by hand, you are running business logic as unversioned text. That is tolerable in a demo, lethal in production. ## Pattern 3: build an evaluation store, not just a test notebook LLM evals are not a one-off event. They are a permanent workflow. You need an evaluation store: a dataset of inputs, expected behaviors, and judgments that you can run on demand. For each candidate change, you should be able to say "run it against these 500 or 5 000 cases and show me what improved, what regressed, and where." This store should not be static. It grows: - From early synthetic examples you wrote yourself.
• From red-team prompts and failure cases discovered by testers.
• From real traffic, sampled and labeled over time. Crucially, the evals must reflect the actual product. If your app is a RAG-driven document assistant, you include questions plus the relevant docs, and you judge groundedness and citation quality. If your app is a code copilot, you evaluate completions on real repositories. Where MLOps often collapses is when teams treat evals as an afterthought. They tweak prompts, swap models, rewire retrieval, and hope general benchmarks will tell them if they broke anything. They will not. Only evals that mirror your real workloads can do that. ## Pattern 4: separate offline experimentation from online governance You need two distinct loops. The offline loop is where you experiment: different prompts, models, retrieval strategies, safety policy why governments care about your gpu cluster loss functions filters. You use your evaluation store and synthetic workloads to compare configurations. This is where notebooks and offline scripts still matter. The online loop is where you run the system under real traffic and enforce rules: - Rate limits and quotas.
• Safety filters and policy checks.
• A/B tests and canary deployments.
• Shadow mode comparisons between old and new configurations. The mistake is mixing the two. If you push half-baked ideas straight into production because "it worked in the notebook," you will spend your weekends cleaning up. A healthy pattern is: - Develop changes offline.
• Run them against evals.
• Deploy them in shadow mode, logging side-by-side outputs.
• If they look safe and better, move to a controlled rollout.
• Only then make them default. This takes more time than a direct edit to a prompt in a live system, but it lets you sleep. ## Pattern 5: log like you are debugging a distributed system, because you are An LLM-backed application is a distributed system: model calls, retrievers, tools, databases, external APIs. When something goes wrong, you need to reconstruct what happened. That means: - Logging every request with a correlation ID that flows through all components.
• Recording model inputs and outputs in a way that respects privacy but still lets you debug.
• Capturing tool calls, retrieval queries, and retrieved documents alongside the final answer.
• Storing model and config versions used for each call. Without this, you cannot answer basic questions: - Why did the model give this answer yesterday but a different one today?
• Did we regress after the last deployment?
• Is this failure due to retrieval, the base model, or a broken tool? In classical MLOps, you often log features and predictions. For LLMs, you log the whole trace. ## Pattern 6: treat safety and policy as code, not as vibes For LLMs, safety is not a separate document in a shared drive. It is part of the runtime. Policies about what the system may or may not say, what data it may access, and what actions it may perform should be expressed in executable form: - Rules that block or transform inputs before they reach the model.
• Classifiers or secondary models that inspect outputs and flag or block unsafe content.
• Permissions and scopes on tools and data sources, checked on every call.
• Ratelimits and anomaly detection for suspicious usage patterns. If your "safety strategy" is a long prompt asking the model to please behave, you are outsourcing governance to an object that is not deterministic and does not actually understand your risk profile. MLOps for LLMs means wiring these policy layers into the core system and versioning them like any other code. When a rule changes, it should show up in your deployment history. When an incident happens, you should be able to see which policy version was active. ## Pattern 7: build for model churn from day one Models change. Providers update frontier APIs. Open-weight models get new releases. You will not be running the same exact architecture a year from now. If you tie your product too tightly to one model's quirks, you trap yourself. Every upgrade becomes a full rewrite. From an operational standpoint, you want: - Indirection: internal abstractions for "generate answer", "classify", "embed" that hide the exact provider.
• Configuration: model choice, temperature, system prompts, tool lists stored in config, not hard-coded.
• Routing: ability to send some traffic to one model and some to another, for comparison and gradual migration. The goal is not to build a baroque multi-cloud fantasy. It is simply to avoid rewriting your system every time you discover that a different model is cheaper, faster, or better for a specific piece of the workflow. ## What this looks like in practice When you walk through a stack that has made it out of the notebook and into stable production, you tend to see the same shape: - A thin API layer exposing clear contracts, not raw prompts.
• A model layer that can route between one or more providers or self-hosted models.
• A retrieval and tool layer with their own versioning and monitoring.
• An orchestration layer that encodes workflows and agent logic as code.
• Safety and policy layers before and after the model.
• Logging, tracing, and evaluation pipelines cutting across all of it. And around the system, you see process: - Regular evaluation runs on a shared benchmark set.
• Change reviews for prompts, tools, and safety rules, not just for ordinary code.
• Shadow deployments and canaries as a default, not an exception.
• Incident reviews where LLM behavior is analyzed like any other component. The technology will keep shifting. Models will keep improving. What will not change is the gap between "works in my notebook" and "survives in production." Filling that gap is the real work of MLOps for LLMs. It is not glamorous. It does not fit in a single demo. But it is the only way to move from clever prototypes to systems that real users can lean on without wondering what will break next.