Eval-Driven Development — Shipping AI You Can Trust

It's a Thursday afternoon, and Priya is about to make the change that will, if she's not careful, quietly break her company's product for the next nine days before anyone notices.

She doesn't know that yet. From where she's sitting, this is a good day. Her customer-support AI — the one that answers billing questions for a few thousand small businesses — has been costing the company real money on the model bill. A new, cheaper model just dropped. It's a third of the price and, on the marketing page, "matches or exceeds" the old one on every benchmark. Priya swaps one line in the config: the model name. She opens the chat, types "How do I update my card on file?", and reads the answer. It's clean. Friendly. Correct. She tries two more questions. Both perfect.

She ships it. The model bill drops by sixty percent. Her manager sends a thumbs-up emoji. For about a week, this looks like the easiest win of the quarter.

Here is what the three questions didn't catch. The new model, it turns out, is slightly more eager to reassure the user. When a customer asks "Can I get a refund on my annual plan?", the old model said, correctly, "Annual plans are refundable within 30 days of purchase." The new model says, warmly and confidently, "Of course — we'd be happy to process that refund for you." It says this to a customer on day 200 of their annual plan. It says it forty times over nine days, to forty different customers, each of whom now believes they're owed money they're not owed. The first Priya hears of it is when the head of support walks to her desk holding a printout and asking why the bot has been promising refunds the company has no intention of honoring.

Nothing crashed. No error was logged. No test went red — because there were no tests. The code did exactly what it was told. The quality drifted, silently, in a direction no human happened to look. And the only reason it took nine days instead of nine minutes to catch is that Priya had no way, when she swapped that model, to ask the one question that mattered: "Is the new one actually better, across the cases I care about — or just better on the three I happened to try?"

This chapter is about building the thing that would have answered that question in nine minutes. Chapter 18 taught you what evaluations are — the metrics, the taxonomy, the LLM-as-Judge mechanics, the bias mitigation. This chapter is about the workflow: how a builder actually lives with evals day to day. How they sit in your repo, run in your CI, gate your merges, and turn "I think this prompt is better" into "I have proof, dated and versioned, that this prompt is better." If Chapter 18 was the theory of the instrument, this is learning to fly with it.

The Core Idea: Evals Are Your Test Suite

Let's name the mental shift, because everything else follows from it.

When you write ordinary software, you write tests. You assert that add(2, 2) returns 4. You run the tests before you merge. If a test goes red, you don't ship — you fix the code or fix the test, but you don't pretend the red doesn't exist. This discipline is so ingrained that shipping without tests feels reckless to most engineers. Nobody argues about whether to have a test suite. They argue about coverage percentages.

AI systems broke this, and they broke it badly, because the thing that makes a test a test — the same input always produces the same output — is exactly the thing LLMs don't give you. Ask Claude to summarize a document twice and you get two different summaries, both fine. assertEqual(summary, "the expected summary") is meaningless. So a whole generation of AI features got built with no test suite at all. Not because the builders were lazy, but because their old tool — exact-match assertions — simply didn't fit the new material.

Evals are the test suite for the non-deterministic. They don't assert exact equality. They assert properties: "the answer contains the correct refund window," "the answer doesn't promise things the policy forbids," "the tone is professional," "the response cites a real source." You run them on a set of inputs you care about. You get a score. And — this is the whole game — you run them before and after every change, so you can see, in numbers, whether the change helped or hurt.

This is the discipline the rest of the chapter installs, in the order you'll actually build it: write a starter eval set, wire up an LLM judge, freeze a golden dataset, gate your CI, measure every change, and watch for drift. That sequence is not arbitrary. It's the path of least resistance from "I have nothing" to "I sleep at night."

Where Eval-Driven Development Lives in the Loop

In Chapter 35 you learned the agentic loop: Spec → Generate → Verify. Eval-driven development is what happens when you take the Verify step seriously enough to automate it and keep it forever.

THE EVAL-DRIVEN LOOP
==============================================================

   +-------------------------------------------------+
   |                                                 |
   v                                                 |
 +-------+    +----------+    +--------+    +-------+ |
 | SPEC  |--->| CHANGE   |--->| RUN    |--->| GATE  |-+
 |       |    | prompt,  |    | EVALS  |    |       |
 | what  |    | model,   |    | golden |    | pass: |
 | good  |    | retrieval|    | set vs |    |  ship |
 | means |    | or code  |    | baseline|   | fail: |
 +-------+    +----------+    +--------+    | back  |
   ^                                        +---+---+
   |                                            |
   |   fail routes to SPEC or CHANGE,           |
   +-- never to "just rerun and hope" ----------+

  The eval set is the written-down form of "what good means."
  Without it, GATE has nothing to check against.

Notice the eval set sits at the top of the loop, not the bottom. It is the executable version of your spec. When someone asks "what does this AI feature need to do?", the honest answer is "whatever the eval set checks for." If your eval set is thin, your definition of done is thin, and you will ship thin. The eval set is not a chore you do after building. It is the building.

Step 1: Write the First Eval Set (The Day-One Move)

The single highest-leverage hour in any AI project is the one where you stop testing in the chat window and write your first ten test cases to a file. Not a hundred. Not a perfect taxonomy of edge cases. Ten. Today.

The reason most teams never do this is that they imagine the finished eval suite — hundreds of cases, a scoring rubric, a CI integration — and the size of that imagined thing scares them into doing nothing. So let's make the first version embarrassingly small. An eval set, at its core, is a list. Each item has an input and some notion of what a good output looks like. That's it.

eval_set.jsonl  (one JSON object per line)
==============================================================
{"input": "How do I update my card on file?",
 "must_contain": ["billing settings", "payment method"],
 "must_not_contain": ["refund", "cancel"]}

{"input": "Can I get a refund on my annual plan?",
 "must_contain": ["30 days"],
 "must_not_contain": ["of course", "happy to process"]}

{"input": "Is my data shared with third parties?",
 "must_contain": ["privacy policy"],
 "must_not_contain": ["sold", "we share everything"]}

Three cases. Look at what they already buy you. The second case — the exact one that bit Priya — now has teeth. must_contain: ["30 days"] and must_not_contain: ["of course"] would have gone red the instant she swapped the model. The nine-day outage becomes a nine-second red mark on her screen, before the change ever leaves her laptop.

The earliest, cheapest form of scoring is the assertion eval: simple string or rule checks, no AI judge involved. "Does the output contain this required phrase? Does it avoid this forbidden one? Is it valid JSON? Is it under 200 words?" These are crude. They miss nuance. They are also free, instant, deterministic, and catch an astonishing number of real regressions. Do not skip them on your way to fancier methods. Many production eval suites are seventy percent dumb assertions and thirty percent AI judgment, and that ratio is correct.

Where do the cases come from? Three wells, in order of value:

Your own head, on day one. You're the domain expert at the start. You know the questions that matter and the answers that are correct. Write them down before you forget them.

Real production logs, forever after. The instant your feature has real users, their inputs are gold — they are, by definition, the actual distribution you must handle, including the weird ones you'd never have invented. Sample them weekly. Every genuinely hard or surprising real query becomes a new permanent eval case.

Every bug, the moment it's found. This is the most important habit in the whole chapter, so it gets its own section below. When Priya's refund bug surfaced, the fix was not just correcting the prompt. The fix was adding that exact question to the eval set so it can never silently return. A bug you've turned into an eval case is a bug that's dead forever. A bug you merely patched is a bug waiting for its sequel.

Step 2: Bring in the Judge (When Rules Aren't Enough)

Assertion evals have a ceiling. "Does the answer contain '30 days'?" is great until the model, correctly, says "within a month." Now your dumb check goes red on a right answer — a false alarm — and you learn the painful lesson that the world's meanings don't live in exact strings.

This is where you graduate to LLM-as-Judge, covered in depth in Chapter 18. The mechanics there still apply — the rubric, the position bias, the verbosity bias, the calibration against humans. What matters here, in the workflow, is when you reach for a judge and how you keep it honest in a pipeline that runs a hundred times a day.

You reach for a judge when the property you care about is semantic, not syntactic. "Is the tone professional?" "Does this answer actually address what was asked, or does it dodge?" "Is this faithful to the retrieved document, or did it embellish?" No string match captures these. A second model, handed a clear rubric, captures them surprisingly well.

ASSERTION EVAL vs. JUDGE EVAL
==============================================================

  PROPERTY                       USE
  ----------------------------   ----------------------
  Valid JSON?                    assertion  (cheap)
  Under 200 words?               assertion  (cheap)
  Contains required disclaimer?  assertion  (cheap)
  Avoids a forbidden phrase?     assertion  (cheap)
  ----------------------------   ----------------------
  Is the tone professional?      judge      (semantic)
  Does it answer the question?   judge      (semantic)
  Faithful to the source doc?    judge      (semantic)
  Better than the old answer?    judge      (pairwise)

  Rule of thumb: reach for the cheap check first.
  A judge you didn't need is latency and cost you
  didn't need. Every judge call is a model call.

The workflow trap with judges is treating their scores as ground truth. They are not. A judge is itself an AI feature, which means — say it with me — the judge needs its own eval set. Before you trust a judge to grade a thousand outputs, hand-grade twenty yourself, run the judge on the same twenty, and check the agreement. If your human "8/10" lines up with the judge's "8/10" most of the time, you can trust it at scale. If it doesn't, your rubric is vague, and you fix the rubric before you fix anything else. A judge you haven't calibrated is a thermometer you haven't checked against ice water — it produces numbers, and the numbers are confident, and you have no idea if they're real.

Step 3: Freeze a Golden Dataset

Once your eval set has grown past a couple dozen trustworthy cases, you do something formal with it: you freeze it into a golden dataset and treat it like a contract.

A golden dataset is a curated, versioned, slow-changing set of inputs with known-good expectations, verified by someone who actually knows the right answer. The word golden is doing real work. These aren't scratch test cases you tweak whenever they're inconvenient. They are the canonical definition of correct for your system, and they change only with deliberation, the way you'd amend a constitution rather than edit a sticky note.

Why the ceremony? Because the golden set is the fixed yardstick against which every future change is measured, and a yardstick that quietly changes length is worthless. If you "improve" your eval set in the same commit where you change your prompt, you can no longer tell whether the score moved because the system got better or because you moved the goalposts. The golden set must hold still while everything else changes around it. That's the entire source of its power.

GOLDEN DATASET: THE FIXED YARDSTICK
==============================================================

  version  what changed       golden set   accuracy
  -------  -----------------   ----------   --------
  v1.0     first prompt        FROZEN v3    82%   <- baseline
  v1.1     reworded system     FROZEN v3    86%   better, proven
  v1.2     cheaper model       FROZEN v3    71%   WORSE, blocked
  v1.3     model + new prompt  FROZEN v3    88%   better, proven
  -------  -----------------   ----------   --------
  v2.0     added 40 new cases  FROZEN v4    --    new baseline set

  The golden set holds still (v3) while the SYSTEM changes.
  Only at a deliberate milestone (v2.0) do you re-freeze the
  yardstick itself -- and you re-baseline everything against it.

Look at row v1.2. That's Priya's change, and against a frozen golden set it shows up as a fat red 71% — a fifteen-point drop — before merge. The golden dataset is the difference between catching that drop in CI and catching it from the head of support holding a printout.

A few hard-won rules for golden datasets:

Cover the distribution, not just the easy middle. A golden set of only golden-path questions tells you the system handles easy cases — which you already knew. Deliberately include the awkward, the adversarial, and the should-refuse. The tail is where reputations are made and lost.

Verify the expectations with a human who knows. A golden case with a wrong "correct answer" is worse than no case at all — it actively trains your system toward the wrong behavior and punishes the right one. The expensive, unglamorous work of having a domain expert sign off on the expected answers is what makes the set golden rather than merely large.

Version it in git, next to the code. The eval set is source. It belongs in the repo, in pull requests, in code review. "Why did we add this case?" should be answerable from the git history, with a link to the bug or the customer complaint that motivated it.

Step 4: The Regression Gate — Evals in CI

Here is where eval-driven development stops being a personal habit and becomes engineering. You take your frozen golden set and you wire it into your continuous integration pipeline, right next to your unit tests, so that no change reaches production without passing through it.

A regression gate is a rule in CI that says: run the golden set against the proposed change; if the score drops below the agreed threshold, the build fails and the merge is blocked. Exactly like a unit test going red blocks a merge. The eval score becomes a first-class citizen of your pipeline, as binding as "the tests pass" and "the code compiles."

THE REGRESSION GATE IN CI
==============================================================

  developer opens pull request
            |
            v
  +------------------------+
  | CI pipeline runs       |
  |  1. lint               |
  |  2. unit tests         |
  |  3. EVAL SUITE  <----- the new citizen
  +-----------+------------+
              |
        score vs. baseline
              |
      +-------+-------+
      |               |
      v               v
  >= threshold    < threshold
   GATE PASS       GATE FAIL
      |               |
      v               v
   merge OK       blocked + report:
                  "accuracy 88% -> 71%,
                   12 cases regressed,
                   here are the 3 worst"

The magic is in that failure message. A regression gate doesn't just say "no." It says which cases regressed and how. A pull request that drops your refund-question accuracy from 96% to 78% arrives with a report naming the twelve newly-broken cases and showing the old answer beside the new one. The reviewer doesn't have to guess what broke. The diff is the explanation. This turns a vague worry ("does this prompt change hurt anything?") into a concrete, reviewable artifact.

Three workflow decisions make or break the gate:

Pick the threshold like an adult. "Zero regressions allowed" sounds noble and is usually wrong — LLM evals have inherent run-to-run variance, so a too-strict gate cries wolf and gets disabled within a week, which is the worst outcome of all. A live gate at 90% beats a perfect gate everyone bypasses. Set the bar where it catches real drops without flagging noise, and tighten it as your suite stabilizes.

Separate the fast gate from the full gate. Running 500 judge-scored cases on every commit is slow and expensive and will make developers hate you. Run a fast subset — fifty cheap assertion cases — on every push, for instant feedback. Run the full golden suite, judges and all, on the merge to main or nightly. Fast feedback where you need speed; deep coverage where you need certainty.

Make the gate's verdict legible to a human. The output of an eval run should be a report a product manager can read — accuracy by category, the worst offenders, a before/after diff — not a wall of raw JSON. The whole point is to make quality visible to the people who decide what ships.

Step 5: Measure Before and After — Every Change

Now we arrive at the daily heartbeat of the practice, the thing you do dozens of times a week once it's wired up. Every time you touch anything that could change the model's behavior, you measure before and after against the frozen set. Every time. No exceptions, no "this one's obviously fine."

What counts as "anything that could change behavior" is broader than people expect:

• A prompt edit. You reworded the system prompt to be clearer. Did clarity for you mean clarity for the model, or did you accidentally break a case that depended on the old phrasing?
• A model swap. The new model is cheaper or newer. Priya's whole disaster lived here.
• A retrieval change. You changed the chunk size or the embedding model in your RAG pipeline (Chapter 13). Retrieval shifts can silently change which documents reach the model, and thus every answer.
• A temperature or parameter tweak. Subtle, easy to forget, perfectly capable of changing your refusal behavior.
• A "tiny" dependency bump. The model provider updated something on their end. You changed nothing in your code and your behavior moved anyway.

BEFORE / AFTER: THE ONLY HONEST COMPARISON
==============================================================

  CHANGE: "rewrite system prompt to be more concise"

                    accuracy  refusal-  tone    cost/
                              correct   (judge) query
  ----------------  --------  --------  ------  ------
  BEFORE (baseline)   91%       98%      4.4     $0.012
  AFTER (new prompt)  93%       81%  <-- 4.6     $0.009
  ----------------  --------  --------  ------  ------
  verdict: accuracy UP, tone UP, cost DOWN...
           but refusal-correct CRATERED 98 -> 81.
           The concise prompt dropped a safety clause.
           NET: do not ship. Fix refusals, re-measure.

Read that table slowly, because it contains the deepest lesson in the chapter. Four of five metrics improved. If Priya — or anyone — had looked only at accuracy, this ships, and a fifth of the inputs that should be refused now slip through. A single number can ship a disaster. Evals earn their keep precisely when they show you the dimension you weren't watching collapsing while the one you were watching climbed. The concise prompt was genuinely better at answering — and genuinely worse at declining — and only a multi-dimensional before/after made that visible before it became a headline.

This is also the answer to one of the most expensive questions in AI engineering: "Should we upgrade to the new model?" Without evals, that's a debate — opinions, vibes, a demo someone cherry-picked. With evals, it's an afternoon. Point your golden suite at the new model, point it at the old one, read the table. The new model wins on cost and loses on faithfulness? Now you're having a real conversation about a real tradeoff, with numbers, instead of arguing about three anecdotes. The eval set turns model selection from religion into measurement.

Step 6: Catch the Silent Drift

Everything so far defends against changes you make. The subtlest failures come from changes nobody makes — the system rotting in place while the code sits perfectly still. This is silent quality drift, and it is the failure mode that separates teams who've run AI in production for a year from teams who haven't yet learned to fear it.

Traditional software doesn't drift. A function that returned the right answer last March returns the right answer today; bits don't rot. AI systems drift because the world the system sits in keeps moving even when the code doesn't:

• The model changed under you. API-served models get updated by their providers, sometimes without a version bump you'd notice. A prompt that reliably produced clean JSON in April starts producing JSON wrapped in markdown fences in May. Your code is byte-for-byte identical. Your output isn't.
• The users changed. Last quarter they asked about onboarding. This quarter, after a product launch, they're asking about a feature your eval set has never heard of. Your scores look fine because your golden set is testing last quarter's questions.
• The knowledge went stale. Your RAG documents describe a pricing tier that was renamed two months ago. The retrieval works perfectly. It perfectly retrieves the wrong, outdated answer.

SILENT DRIFT: SAME CODE, FALLING SCORES
==============================================================

  golden-set accuracy, measured weekly, code UNCHANGED:

  week 1  ########################################  94%
  week 2  #######################################   93%
  week 3  ######################################    92%
  week 4  ###################################       88%  <- ?
  week 5  ##############################            78%  <- ALERT
          |
          +-- nobody deployed anything.
              the model provider shipped a silent update
              on week 4. only the weekly eval run saw it.

  A 5% drop over a month = investigate.
  A 10% drop in a day    = incident, page someone.

The only defense against drift is to run your evals on a schedule, against live or sampled production traffic, whether or not anyone changed anything. A nightly or weekly run of the golden set, plus a sample of recent real inputs, and an alert wired to a downward trend. That's it. It is unglamorous infrastructure that does nothing visible for weeks at a time — and then, one Tuesday, it catches the provider's silent update on day one instead of day nine, and pays for its entire existence in a single alert.

Putting It Together: A Week in the Practice

Let's watch the whole discipline run for one week, so the pieces connect into a rhythm rather than a list.

Monday. A product manager asks for the support bot to handle questions about a brand-new feature. You write five new golden cases first — before touching the prompt — capturing what correct answers look like. Definition of done, written down. Then you edit the prompt to cover the feature. You run the full golden set locally: the five new cases pass, and — critically — the eighty old ones still pass. No regression. You open a pull request.

Tuesday. CI runs your eval gate on the PR. Accuracy holds at 92%, refusal-correctness holds at 97%, the five new cases are green. The gate passes. A teammate reviews the before/after report attached to the PR, sees no category regressed, and approves. You merge. The whole "is this safe to ship?" question took the length of a coffee break and produced a paper trail.

Wednesday. The nightly drift run flags that faithfulness dropped four points overnight. Nobody deployed. You investigate, and find the model provider pushed an update that made the model slightly chattier, adding unsourced sentences to RAG answers. You tighten the "use only the provided context" instruction in the prompt, measure before/after — faithfulness back up, nothing else hurt — and ship the fix. A drift that would've taken a week to surface as complaints was a one-day, instrumented non-event.

Thursday. A customer reports a genuinely new failure: a phrasing of a question your bot mishandles. You don't just patch it. You add that exact phrasing to the golden set with its correct answer, then fix the prompt, then confirm the new case goes green and nothing else broke. The bug is now immortalized as a test. It cannot silently return.

Friday, 5 p.m. Someone wants to ship the cheaper model before the weekend to save on costs. A quarter ago this would've been a "looks fine, send it" — Priya's exact trap. Instead, you point the golden suite at the cheaper model. It comes back fifteen points down on refusal-correctness. The gate would have blocked it anyway, but you didn't even need the gate to fight the argument — the table settles it in three minutes. The model stays unshipped, the weekend stays calm, and the discipline that "slowed you down" all week just saved you from a nine-day outage you'll never even know you avoided.

That last point is the quiet tragedy and the quiet triumph of eval-driven development: its biggest wins are invisible. You never see the headlines you didn't generate. You never get thanked for the outage that didn't happen. The reward for doing this well is a series of uneventful Fridays — which, to anyone who has lived through an eventful one, is the most underrated luxury in software.

The Practitioner's Checklist

If you're building, leading, or evaluating an AI feature, these are the questions that separate teams flying with instruments from teams flying on hope:

On the eval set: Does it exist as a file, in version control, or does "testing" still happen in a chat window? How many cases? Do they cover the should-refuse and the expensive-if-wrong, or only the easy middle? When was the last one added, and what motivated it?

On the judge: If you use an LLM judge, has it been calibrated against human grades on a real sample — or are you trusting its confident numbers blind? Does the judge share a model family with the thing it's judging (the circular-judge trap)?

On the golden set: Is there a frozen, human-verified yardstick that holds still while the system changes? Or does the eval set get "improved" in the same commits that change the prompt, quietly moving the goalposts?

On the gate: Do evals run in CI and block merges, or are they a thing someone runs manually when they remember? Is the threshold set where it catches real drops without crying wolf? Can a non-engineer read the gate's report?

On before/after: When you change a prompt or swap a model, do you measure across multiple dimensions — or could a single climbing number be hiding a collapsing one? Could you settle "should we upgrade the model?" with a table this afternoon, or only with a debate?

On drift: Do evals run on a schedule, against production traffic, whether or not anyone changed anything? How would you find out — and how fast — if quality silently fell 10% next week with zero code changes? From a graph, or from a churn report?

The teams that wire these six things into their workflow ship AI they can trust, and they sleep through their Fridays. The teams that skip them ship AI they hope works, and they learn about their failures the way Priya did — from a colleague holding a printout, nine days too late. In deterministic software, "I tested it and it works" was a reasonable thing to say. In AI, it's a sentence that should make you nervous. Evals are how you replace it with the only thing that holds up under scale, under malice, and under the cold light of six months from now: "I have evidence."

Chapter endnotes

[1] This chapter builds directly on the evaluation concepts introduced in Chapter 18 — the evaluation taxonomy, LLM-as-Judge mechanics, position and verbosity bias, and the Hamel Husain "vibes → quantitative → automated" maturity model. Where Chapter 18 explains what to measure and how the metrics behave, this chapter covers the development workflow of living with evals: writing eval sets, freezing golden datasets, gating CI, and monitoring drift. Read them as a pair.

[2] The framing of "evals as the test suite for non-deterministic systems" reflects a now-common practice in AI engineering teams, popularized in part by Hamel Husain's writing at hamel.dev and by the broader "evals are all you need" sentiment among practitioners shipping LLM products. The key shift — from exact-match assertions to property-based and judge-based scoring — is the practical adaptation of test-driven development to systems whose outputs vary by design.

[3] The distinction between assertion-based evals (cheap, deterministic, syntactic) and judge-based evals (expensive, semantic) mirrors the layered testing pyramid familiar from traditional software, where fast cheap unit tests outnumber slow expensive integration tests. The recommended bias toward cheap assertions first — reaching for an LLM judge only when a property is genuinely semantic — is a cost-and-latency discipline learned repeatedly by teams whose first eval suites were all judge calls and whose model bills reflected it.

[4] The regression-gate pattern — running an eval suite in CI/CD and blocking merges on score regressions against a frozen baseline — is the natural extension of the continuous-evaluation practice described in Chapter 18's eval pipeline. Tooling in this space (eval frameworks integrated with CI providers) matured rapidly through 2024–2026; the core idea, however, is independent of any tool: a versioned golden dataset, a threshold, and a pipeline that enforces it.

[5] The "silent drift" failure mode — degrading quality with no corresponding code change — is one of the defining differences between operating AI systems and operating traditional software. Its causes (silent provider model updates, shifting user-input distributions, staling knowledge bases) are discussed in Chapter 18 under "Monitor Drift"; this chapter's contribution is the operational prescription: scheduled evals against sampled production traffic, with alerting on downward trends, as the only reliable detector.

[6] The PrabhupadaAI golden-dataset example (200 scholar-validated question-answer pairs) is the same system introduced in Chapter 13's discussion of RAG over spiritual literature. Its evaluation effort — a large fraction of total project work spent on building and maintaining the validated test set — is representative of well-run AI projects generally, where the unglamorous work of defining and verifying "correct" is the work that most directly determines whether the system can be trusted.