Free lesson · AI Engineering
Why AI Evaluation Is Different
Why AI Evaluation Is Different
The Test Suite That Passed While the Product Failed
At Meta, we had a content classification service backed by an LLM. The test suite contained 847 unit tests. Every test asserted a specific input against a specific expected output. Every test passed. The CI pipeline was green. We shipped.
Within 72 hours, 14% of user-reported content moderation appeals were being upheld - meaning the classifier was wrong on 14% of escalated cases. The test suite said the system was working. Production said it was not. Both were telling the truth, and understanding why is the first thing every AI engineer needs to grasp about evaluation.
The tests were deterministic assertions against cached model outputs. When we wrote the tests, we called the model, captured the response, and hardcoded it as the expected value. The tests did not test the model. They tested whether the caching layer returned the cached output correctly. The actual model - the live, non-deterministic, prompt-sensitive, temperature-dependent model - was never evaluated by the test suite. It could have been replaced with a random number generator and the tests would still have passed, because the tests were not measuring what mattered.
This is the fundamental problem. Engineers arriving at AI systems from deterministic software bring a testing mental model built for functions with fixed outputs. That mental model is necessary - you still need unit tests for your data pipelines, your API handlers, your caching logic - but it is radically insufficient for the AI components. AI evaluation is a different discipline with different tools, different metrics, and different failure modes.
What Makes AI Systems Untestable With Traditional Methods
Traditional software testing rests on three assumptions that AI systems violate.
Assumption 1: Deterministic Outputs
A function add(2, 3) returns 5 every time. You write assert add(2, 3) == 5 and that assertion is valid forever, on every machine, in every environment. The output is a deterministic function of the input.
An LLM call summarise("quarterly earnings report") returns a different summary every time. Even at temperature 0, subtle differences arise from floating-point arithmetic, GPU batching, and model versioning. You cannot write assert summarise(report) == "Revenue grew 12%..." because the next call might phrase it differently, emphasise different points, or structure the summary differently - all while being equally correct.
The implication: Exact-match assertions are useless for evaluating AI outputs. You need evaluation methods that assess quality on a spectrum rather than checking equality against a single expected value.
Assumption 2: Finite Output Space
A function that returns a boolean has two possible outputs. A function that returns an HTTP status code has roughly 60. You can enumerate the output space and write tests for each meaningful case. The testing is exhaustive in principle, even if you choose to test only a subset.
An LLM that generates text has an effectively infinite output space. A 200-token response drawn from a 100,000-token vocabulary has more possible outputs than atoms in the observable universe. You cannot enumerate the output space. You cannot test all cases. You can only sample, measure, and make probabilistic statements about quality.
The implication: Test coverage as a concept does not translate. Instead of "what percentage of outputs have we tested," the question becomes "how representative is our evaluation dataset of the inputs the system will see in production."
Assumption 3: Stable Behaviour Over Time
Once you deploy a deterministic function, its behaviour does not change unless you change the code. A test that passes today passes tomorrow, next month, and next year, as long as the code is unchanged.
An AI system's behaviour changes without any code changes. The model provider updates the model. The user population shifts, sending different distributions of inputs. The world changes, making the model's training data less representative of current reality. The evaluation that said the system was 92% accurate in January may find it is 84% accurate in June - not because anything in your codebase changed, but because the world around the model moved.
The implication: Evaluation is not a one-time gate. It is a continuous process that must run in production, on live traffic, indefinitely. The "test once, ship, forget" workflow does not survive contact with AI systems.
graph TD
subgraph Traditional["Traditional Software Testing"]
T1["Deterministic outputs\nassert f(x) == y"]
T2["Finite output space\nenumerate and test"]
T3["Stable over time\ntest once, ship"]
T4["Green CI = safe to ship"]
end
subgraph AI["AI System Evaluation"]
A1["Non-deterministic outputs\nassess quality on a spectrum"]
A2["Infinite output space\nsample and measure statistically"]
A3["Behaviour drifts over time\nevaluate continuously in production"]
A4["Green CI = necessary but not sufficient"]
end
T1 -.->|"Violated by"| A1
T2 -.->|"Violated by"| A2
T3 -.->|"Violated by"| A3
T4 -.->|"Becomes"| A4
style Traditional fill:#1e3a5f,stroke:#7dd3fc,stroke-width:2px
style AI fill:#1e1b4b,stroke:#818cf8,stroke-width:2px
This diagram contrasts traditional software testing with AI system evaluation across four dimensions. Traditional testing assumes deterministic outputs, finite output spaces, stable behaviour over time, and that green CI means safe to ship. AI evaluation replaces these with non-deterministic outputs assessed on a quality spectrum, infinite output spaces sampled statistically, behaviour that drifts requiring continuous production evaluation, and green CI being necessary but not sufficient for shipping.
The Evaluation Mindset Shift
Engineers moving from traditional software to AI systems need to make three mental model shifts. These shifts are uncomfortable because they replace certainty with probability, binary with spectrum, and finite with ongoing.
Shift 1: From "Correct / Incorrect" to "Quality on a Spectrum"
In traditional testing, an output is correct or incorrect. There is no middle ground. add(2, 3) returns 5 (correct) or it does not (incorrect).
In AI evaluation, most outputs exist on a quality spectrum. A summary might capture all key points but use awkward phrasing (quality: 0.8). Another summary might have elegant prose but miss one key point (quality: 0.75). A third might capture everything perfectly but include one sentence that contradicts the source (quality: 0.6 - because the contradiction matters more than the correct parts). None of these are "incorrect" in the binary sense. All of them have measurable quality that can be compared and improved.
What this means in practice: You need evaluation metrics that return numbers, not booleans. "Faithfulness: 0.87" tells you something actionable. "Test: PASS" tells you nothing about whether the output is good enough for production.
Shift 2: From "Test Coverage" to "Evaluation Representativeness"
In traditional testing, you aim for high coverage - what percentage of code paths have tests? 80% branch coverage is a common target.
In AI evaluation, the equivalent question is: how well does your evaluation dataset represent the distribution of inputs your system will see in production? A 1,000-example evaluation set that covers 95% of input patterns tells you more than a 10,000-example set that covers only 60% of patterns. The quality of the evaluation dataset - its representativeness, its difficulty distribution, its coverage of edge cases - matters more than its size.
What this means in practice: Building the evaluation dataset is as important as building the model. Most teams underinvest in evaluation datasets because they feel like testing artifacts rather than product assets. They are product assets. A bad evaluation dataset gives you false confidence. False confidence is worse than no confidence at all.
Shift 3: From "Gate Before Ship" to "Monitor After Ship"
In traditional software, testing is a gate: the tests pass, you ship. The system's behaviour is stable after shipping, so the pre-ship gate is sufficient.
In AI systems, pre-ship evaluation is necessary but the real evaluation happens after shipping, on production traffic, continuously. The distribution of inputs changes. The model provider updates the model. User expectations evolve. A system that met all quality bars at ship time can silently degrade without any deployment, any code change, or any human decision. Only continuous production evaluation catches this degradation.
What this means in practice: You need two evaluation systems: an offline evaluation pipeline that runs before deployment (the gate), and an online evaluation system that runs continuously in production (the monitor). Most teams build only the first. The teams that build both are the ones whose AI features remain healthy over time.
The Four Layers of AI Evaluation
A production AI system needs evaluation at four layers, each catching different types of problems.
Layer 1: Component Tests (Traditional)
Unit tests for the non-AI components: data pipelines, API handlers, caching logic, prompt template construction, response parsing. These are standard engineering tests. They use exact-match assertions, mock external dependencies, and run in CI. They are fast, cheap, and essential - but they test the scaffolding, not the intelligence.
Layer 2: Offline Evaluation (AI-Specific)
Evaluation of the AI components against a curated dataset before deployment. This is where AI-specific metrics (faithfulness, relevance, coherence) are measured. The evaluation dataset should be large enough for statistical significance (typically 200+ examples for reliable confidence intervals) and representative of production traffic. Offline evaluation catches regressions in prompt quality, model changes, and pipeline logic errors.
Layer 3: Online Evaluation (Production)
Evaluation of the AI system on live production traffic. This includes automated quality sampling (run the evaluation pipeline on a random 5-10% of live requests), user feedback analysis (thumbs up/down, regeneration rates, edit rates), and A/B testing (compare new prompts or models against the current version). Online evaluation catches the problems that offline evaluation misses: distribution shift, edge cases not in the eval set, and quality degradation over time.
Layer 4: Human Evaluation (Ground Truth)
Periodic human evaluation provides the ground truth that automated metrics are calibrated against. Without human evaluation, you cannot know whether your automated metrics actually correlate with quality. Human evaluation is expensive and slow, which is why it is periodic rather than continuous - but it is the anchor that keeps the entire evaluation system honest.
graph TD
subgraph Layer1["Layer 1: Component Tests"]
L1["Unit tests, integration tests\nExact-match assertions\nFast, cheap, CI-integrated"]
L1C["Catches: pipeline bugs,\nAPI errors, parsing failures"]
end
subgraph Layer2["Layer 2: Offline Evaluation"]
L2["Curated eval dataset\nAI-specific metrics\nPre-deployment gate"]
L2C["Catches: prompt regressions,\nmodel quality changes"]
end
subgraph Layer3["Layer 3: Online Evaluation"]
L3["Production traffic sampling\nUser feedback analysis\nA/B testing"]
L3C["Catches: distribution shift,\nedge cases, quality drift"]
end
subgraph Layer4["Layer 4: Human Evaluation"]
L4["Expert review, rubric scoring\nInter-rater reliability\nPeriodic ground truth"]
L4C["Catches: metric miscalibration,\nsubtle quality issues"]
end
Layer1 --> Layer2
Layer2 --> Layer3
Layer3 --> Layer4
Layer4 -.->|"Calibrates"| Layer2
Layer4 -.->|"Calibrates"| Layer3
style Layer1 fill:#1e3a5f,stroke:#7dd3fc,stroke-width:2px
style Layer2 fill:#312e81,stroke:#a5b4fc,stroke-width:2px
style Layer3 fill:#854d0e,stroke:#fde047,stroke-width:2px
style Layer4 fill:#166534,stroke:#86efac,stroke-width:2px
This diagram shows four evaluation layers stacked from bottom to top. Layer 1 is Component Tests - unit and integration tests with exact-match assertions, catching pipeline bugs and API errors. Layer 2 is Offline Evaluation - curated eval datasets with AI-specific metrics, catching prompt regressions and model quality changes. Layer 3 is Online Evaluation - production traffic sampling and A/B testing, catching distribution shift and quality drift. Layer 4 is Human Evaluation - expert review with rubric scoring, catching metric miscalibration and subtle quality issues. Human evaluation calibrates both the offline and online layers through feedback loops.
What This Module Covers
This module dedicates one topic to each evaluation challenge, in the order you will encounter them when building a production AI system.
Topic 13.2 covers building the evaluation datasets that Layers 2 and 3 depend on. Without good data, no evaluation method works. Topic 13.3 covers the offline evaluation frameworks - the tools and metrics for Layer 2. Topic 13.4 covers LLM-as-judge evaluation - using one model to evaluate another, which is the most scalable automated evaluation approach but has failure modes you need to understand. Topic 13.5 covers human evaluation - the Layer 4 ground truth that keeps automated metrics honest. Topic 13.6 covers online evaluation and A/B testing - how to evaluate in production without breaking things. Topic 13.7 covers continuous evaluation - building evaluation into the production loop so quality monitoring is automatic and ongoing.
The progression is deliberate: build the dataset first, then learn to evaluate offline, then learn to evaluate with LLMs, then calibrate with humans, then evaluate in production, then make it continuous. Each topic builds on the previous one.
Interview: System design interview connection: "How would you test this AI feature?" is a question that immediately separates candidates who have built production AI systems from those who have not. The weak answer is "write unit tests." The strong answer describes the four evaluation layers - component tests for the scaffolding, offline evaluation for pre-deployment quality gating, online evaluation for production monitoring, and periodic human evaluation for ground truth calibration.
What to emphasise: Lead with the three assumptions that AI violates (non-deterministic, infinite output space, behaviour drift). Then describe the four-layer evaluation architecture. Then explain why continuous evaluation matters - a system that passes all quality gates at ship time can silently degrade without any code change.
What separates strong from weak answers: Weak answers treat evaluation as a pre-ship gate (Layer 2 only). Strong answers describe the full evaluation lifecycle: build an eval dataset, run offline evaluation before deployment, run online evaluation continuously in production, and periodically recalibrate with human evaluation. The strongest answers mention specific metrics (faithfulness, relevance, coherence) and acknowledge that those metrics must be calibrated against human judgment.
Exercise: Starter: You have a customer support chatbot that uses RAG to answer questions from a knowledge base. Your team lead asks: "Do we need tests for this?" Describe the four evaluation layers you would build. For each layer, give one specific example of a problem it would catch that the other layers would miss.
Intermediate: Your team has a 500-example evaluation dataset for a document summarisation feature. The offline evaluation shows 89% faithfulness. The feature ships, and within two weeks, user feedback indicates that summaries for legal documents are frequently inaccurate - but your evaluation dataset contains only 12 legal documents. Diagnose the evaluation failure. How would you restructure the dataset to prevent this in the future?
Advanced: Design the complete four-layer evaluation architecture for an AI-powered code review tool that reviews pull requests and suggests improvements. For each layer, specify: what you evaluate, what metrics you use, how often you run the evaluation, and what action you take when the evaluation fails. Pay special attention to the calibration loop between Layer 4 (human evaluation) and Layer 2 (offline evaluation) - how do you ensure your automated metrics actually correlate with what developers find useful?
Key Takeaways
AI evaluation is a fundamentally different discipline from traditional software testing. Three assumptions that traditional testing relies on - deterministic outputs, finite output space, and stable behaviour over time - are violated by AI systems. This does not mean traditional tests are useless - component tests remain essential for the non-AI scaffolding - but they are radically insufficient for evaluating the AI components.
The mindset shift from deterministic testing to AI evaluation involves three changes: from correct/incorrect to quality on a spectrum, from test coverage to evaluation representativeness, and from gate-before-ship to continuous monitoring after ship. Each shift replaces comfortable certainty with uncomfortable probability.
Production AI systems need four evaluation layers: component tests (traditional, fast, cheap), offline evaluation (AI-specific metrics against curated datasets), online evaluation (production traffic sampling, user feedback, A/B testing), and periodic human evaluation (ground truth calibration). Most teams build only the first two layers. The teams whose AI features remain healthy over months and years are the ones that build all four.
This module is dedicated to evaluation because evaluation is a discipline, not a footnote. The quality of your evaluation infrastructure determines the quality of your AI product - not because evaluation makes the model better, but because evaluation tells you when the model stops being good enough, before your users tell you.