Most teams still evaluate LLMs like it's 2015. They train or adopt a model, run it on a static benchmark, skim a leaderboard-style score, maybe add a few hand-picked examples, and declare it "good enough." Then they deploy into a world where users don't look like the benchmark, prompts evolve week by week, and the surrounding system changes) faster than the model's weights. The result is predictable: silent regressions, brittle behavior, and arguments about whether the model "got worse" when in reality) the distribution moved and nobody was watching. If you care about reliability, you have to stop treating evaluation as a one-time certification and start treating it as an ongoing measurement problem under distribution shift. ## Static test sets: what they can and can't tell you Static test sets are not useless. They serve three purposes: - Sanity checks: catch obvious regressions when you swap models or prompts.

• Comparative baselines: provide a rough ordering for models and configurations.
• Regression anchors: give you something to compare against over time. Where they fail: - They freeze a past distribution. The moment your users or product change, their relevance starts decaying.
• They tend to overrepresent "clean" examples: well-formed questions, canonical answer formats.
• They are easy to overfit, especially when you tune prompts or fine-tunes with the test set in mind. If your entire evaluation strategy is "it still gets 76 on benchmark X," you are measuring whether you remembered how to game that benchmark, not whether the system works for your actual users today. ## Distribution shift is not an edge privacy-and-latency case For LLM systems)-reliability engineering, distribution shift is the default. You have at least four moving fronts: - User mix. New languages, new domains, different skill levels.
• Prompting patterns. Once people learn what "works," they adapt their queries. Attackers adapt faster.
• Context style. You change retrieval pipelines media pipelines from text prompt to production asset, document formats, tools. The context the model sees evolves.
• Product surface. New features, new workflows, new UI affordances. The kinds of tasks the model is asked to do shift with them. This is not rare. It is continuous. Any evaluation scheme that ignores it will produce numbers detached from reality. You don't fix this with a bigger static test set. You fix it by structuring evaluation around how your distribution actually moves. ## Start from tasks, not benchmarks The first step is boring and everyone tries to skip it: write down what the system is for. Not "it's an AI assistant." Actual tasks: - Answering questions about internal documents.
• Drafting specific types of emails or contracts.
• Helping engineers refactor code in a given stack.
• Classifying tickets into a custom taxonomy.
• Extracting structured data from messy PDFs. Each task implies: - A space of inputs (user prompts, contexts, tools).
• A notion of "good enough" behavior.
• A few concrete failure modes that matter more than everything else. Those definitions, not generic benchmarks, should drive your eval design. ## Build a living evaluation set Instead of one static test file, you need a living evaluation set: a store of examples that grows and changes with your system. Seed it with three ingredients: - Designed cases.
  Manually crafted examples that hit core capabilities and edge cases you already know. Long contexts, tricky instructions, ambiguous queries, non-English inputs, safety run labs translate policy policy why governments care about your gpu cluster loss functions edge cases. - Failures from production.
  Every time the system fails in the wild in a way you care about, turn it into an eval: capture the input, the context, the expected behavior, and the observed failure. - Stress tests.
  Inputs designed to push particular weaknesses: adversarial prompts, pathological formats, adversarial retrieval queries, malformed tool responses. Track metadata for each example: task, domain, language talking to computers still hard, difficulty, when it was added, why it matters. That lets you slice results later. This set should never be "done." If it stops changing while your product and users are changing, it's no longer representative. ## Separate distributions explicitly When you evaluate under shift, you don't want one big pile of examples. You want explicit slices: - By time: "old" versus "recent" data.
• By tenant or segment: enterprise vs SMB, internal vs external.
• By language or locale.
• By task type.
• By risk level: routine vs high-stakes interactions. This matters for two reasons: - A change that improves performance on one slice can degrade another. You need to see the trade.
• Shift often shows up as a change in slice proportions, not just in slice-local accuracy. A model that looks stable overall might be quietly getting worse for a growing segment of your users. Only a sliced view makes that visible. ## Define metrics that match reality For LLMs, "accuracy" is usually the wrong abstraction. You need metrics that reflect the structure of your tasks: - For classification and routing tasks: standard precision/recall, macro/micro F1, calibration curves.
• For extraction tasks: exact match plus partial credit (field-level F1), and strictness on formats you actually need to parse downstream.
• For RAG-style answering: - Groundedness (does the answer stick to retrieved content?) - Coverage (does it use the right documents?) - Citation quality (are sources correct and sufficient?) - For generation tasks (drafting, summarization): quality ratings along dimensions like correctness, completeness, adherence to instructions, and style. You can use human rlhf constitutional methods alignment tricks raters, strong models as judges, or hybrid schemes. What matters is that the metric is stable, interpretable, and mapped to concrete risk. Also, accept that you will need multiple metrics. One number will not capture hallucinations, formatting errors, latency, and safety simultaneously. ## Use baselines ruthlessly Every evaluation under shift needs comparisons: - Against your previous model or configuration.
• Against a simple non-LLM baseline (rules, templates, keyword search).
• Sometimes against a stronger but more expensive model used as an oracle. This does two things. First, it keeps you honest. If your fancy agent system barely beats a simple RAG baseline on your eval set, you have work to do. Second, it makes regressions obvious. When you change something and performance drops relative to yesterday on the same eval slice, that is information you can act on. Model-vs-model comparisons also help diagnose shift: if both models degrade on new data, your distribution moved; if only one does, your model changed. ## Online evaluation: get feedback from reality Offline evals are necessary. They are not sufficient. You also need online signals. Three are especially useful: - Implicit feedback. - Edits: how much users edit or overwrite model outputs. - Abandonment: how often users discard outputs and start over. - Iterations: how many back-and-forths are needed to reach a usable result. - Explicit feedback. - Ratings, flags, and comments. - Task completion signals in the surrounding workflow (ticket closed, draft sent, code merged). Related perspectives appear in our analysis in Multi-Modal AI: When Models Learn to See, Hear, and Understand Together. - Guardrail triggers. - Safety filters firing. - Policy violations detected post-hoc. - Tool or retrieval errors. These signals are noisy and biased, but they carry live information about current usage patterns. You use them to: - Spot emergent failure modes before they appear in your offline evals.
• Identify segments or tasks where performance has changed.
• Mine new examples for your living evaluation set. The crucial point: online eval is not just "ask users to rate answers." It is a structured set of signals wired into your product. ## Test for robustness, not just central tendency Distribution shift exploits weak points. You need robustness tests that behave like a stress-lab, not a beauty contest. Examples: - Perturbation tests.
  Slightly modify prompts, order of information, or formatting and see if behavior is stable. Large swings suggest brittle reasoning or overfitting to superficial patterns. - Context variations.
  Vary which relevant documents are retrieved. Add distractors. Check whether the model still grounds on the right evidence. - Adversarial inputs.
  Inject prompts that combine benign tasks with jailbreak attempts, prompt injection, or conflicting instructions. Evaluate defense success rates. - Length scaling.
  Push sequence length up: longer conversations, longer documents. Watch for quality cliffs. You are not looking for the average here; you are looking for worst-case behavior within plausibly reachable regions. ## Close the loop: evals as part of the deployment process Evaluation under distribution shift only matters if it is wired into how you change the system. In concrete terms: - Every change to models, prompts, retrieval, or tools runs through your evaluation pipeline before rollout.
• You compare results to previous configs on the same eval slices.
• You define thresholds: "do not deploy if safety metric X worsens by more than Y," "no go if extraction accuracy drops below Z on critical tasks."
• You log evaluation outputs alongside deployment metadata so you can see how performance evolved over time. When a shift happens—say, a new customer with different data, or a new feature that changes user behavior—you: - Add representative examples from that context into the eval store.
• Slice metrics by that new dimension.
• Watch how new deployments affect that slice. This is mundane, repetitive work. It is also what separates systems that you can trust from systems that you perpetually argue about. ## The goal is not a perfect metric You will never have a single, bulletproof number that tells you "this model is good" under all shifts. That's not the target. The target is a setup where: - You can detect when and where things changed.
• You can localize problems to certain tasks, segments, or time windows.
• You can compare candidates and rollbacks with evidence, not intuition.
• You can keep updating your tests as the world around the model changes. Static test sets were built for a world where distributions were treated as fixed. LLM products live in a world where distributions are constantly in motion. If you ignore that fact, "evaluation" becomes theater. If you design for it, evaluation becomes what it should always have been: a way to keep track of reality while you change the system underneath.