Building an eval suite for a system that shipped without one

The single most common finding in our production readiness audits is not a bad prompt, a wrong model, or a broken retrieval pipeline. It is a system with no automated evaluation at all. Not a thin suite. None. The deploy ritual is somebody pasting their three favorite prompts into a staging environment, eyeballing the output, and shipping.

That is not a competence problem. Under demo pressure it is a rational trade — evals produce nothing you can show a stakeholder. But once the system is live, every prompt tweak, every model upgrade, and every provider deprecation becomes a coin flip where the test suite is your customers.

This post is the retrofit path we walk on audit engagements: how to take a system that shipped without evals and get it to a suite that gates deploys — without pausing the roadmap to do it. The first useful checks exist by the end of day one. The full progression takes about two weeks of one engineer's part-time attention.

Why systems ship without evals

Three things stop teams, and only one of them is real.

The first is the nondeterminism excuse: you cannot unit-test a model that gives a different answer every time. This is half true and it stops teams completely. The half that is false matters more — most of what an LLM system does is not free-form prose. It extracts fields, calls tools, routes tickets, cites documents. Those have right answers, and right answers can be asserted the boring way.

The second is demo economics. An eval suite is invisible in a sales call, so it loses every prioritization fight until the first production incident, at which point it wins all of them at once.

The third — the real one — is the cold start. Faced with a blank file named evals/, nobody knows what to assert. The way out is to stop trying to write a test set and start mining one.

Step zero: mine your traces for a test set

A system that has been in production for three months has already generated a better test set than anything you could invent in a workshop. Pull 50 to 100 real requests from your logs and make that the backbone of the suite. The mix we aim for:

Roughly 60% routine traffic. The requests the system handles all day. These become your regression floor — the cases that must never break.
Roughly 20% known-hard cases. The long documents, the ambiguous questions, the inputs in the format nobody expected. Every team knows where these live.
Roughly 20% actual failures. Go through the support tickets and the complaints channel. Every answer a customer flagged as wrong is a test case with the label already attached.

Then label them, lightly: pass or fail, plus one line on why. A spreadsheet is fine. Half a day with someone who knows the domain is enough, and that half-day of labels is the most valuable artifact in the whole retrofit — it is what every later level calibrates against.

If you cannot do this because nothing was logged, that is the finding. Stop here and fix tracing first — we covered what that takes in our post on production agent patterns. An eval suite built on invented examples tests the system you imagine, not the one you run.

Progression

The eval retrofit ladder

Hard assertions first, replayed real traffic second, an LLM judge last — and only after it agrees with your human labels.

Level 1: deterministic checks

Start with hard assertions, because they are the cheapest to write and the only ones you never have to argue with. The rule is simple: anything with a right answer gets a deterministic check, never a judge. Schema validity, exact extracted values, numeric ranges, must-contain and must-not-contain strings, whether the model abstained when it should have, latency and cost ceilings.

def test_invoice_extraction(case):
    result = extract_invoice(case.document)

    # Schema first: if this fails, nothing else matters.
    InvoiceSchema.model_validate(result)

    # Exact fields are exact. No judge required.
    assert result.total == case.expected_total
    assert result.currency in {"USD", "EUR", "GBP"}

    # Missing data must produce abstention, not a guess.
    if case.vendor_missing:
        assert result.vendor is None

    # Cost and latency are product features too.
    assert case.trace.total_tokens < 12_000
    assert case.trace.latency_ms < 8_000

Teams consistently reach for an LLM judge first because it feels like the AI-native move. It is the wrong order. A judge asked is this JSON valid? is an expensive random number generator; a schema validator is free and right every time. On most systems we audit, ten checks like the ones above exist by the end of the first day, run on every commit, finish in under a minute, and cost almost nothing — and they already catch the worst class of regression: the confident, malformed, silently wrong output.

Level 2: golden traces

Deterministic checks validate outputs. The next level validates behavior. Take about fifty of the mined requests and snapshot the full trajectory each one produces today: which tools were called, in what order, what the retrieval step returned, what the final answer said. These are your golden traces. On every meaningful change — prompt edit, model upgrade, retriever swap — replay them and diff.

The trap here is asserting exact equality on the output text. That breaks on the first temperature wobble and the team quietly deletes the suite. Assert invariants of the trajectory instead:

def test_refund_workflow_trace(golden):
    trace = replay(golden.request)

    # The right tool, a bounded number of steps.
    assert "lookup_order" in trace.tools_called
    assert len(trace.steps) <= golden.step_budget

    # Retrieval must surface the doc that holds the answer.
    assert golden.policy_doc_id in trace.retrieved_ids

    # The answer must cite it — not free-associate.
    assert trace.final_answer.cites(golden.policy_doc_id)

Invariants survive rewording; they do not survive the system actually getting worse. This is the level that catches the regression class that hurts the most in practice — the prompt change made for one workflow that quietly breaks a different one nobody was looking at.

Level 3: a judge you can actually trust

Only now does LLM-as-judge enter, and only for the qualities you genuinely cannot hard-code: is the answer grounded in the retrieved context, is the tone right for a customer, is the summary faithful to the source. Used this way a judge is invaluable. Used before levels 1 and 2 it is a noise machine that emits numbers.

The difference between the two is calibration, and the loop is short:

Take 30 to 50 outputs your team has already labeled — step zero bought you these.
Write the judge prompt as a rubric with binary verdicts. Pass or fail, with a required one-line reason. Numeric 1–10 scores feel more rigorous and are mostly noise.
Run the judge over the labeled set and measure agreement with the humans. Below about 85–90%, the judge is not measuring what you think it is — tighten the rubric and rerun, and keep going until it agrees.
Pin the judge's model version, and re-calibrate whenever the rubric, the judge model, or the shape of your traffic changes.

Once the rubric is tight, a small, cheap model is almost always enough to run it. Frameworks like DeepEval give you the scaffolding for free, but the scaffolding was never the hard part. The hard part is the labeled set the judge answers to — which is why step zero comes first.

Wire it into the deploy path

An eval suite nobody is forced to run is documentation. The retrofit only holds if the suite gates the deploy, so the last step is wiring it into CI with three rules:

Level 1 is binary. Every deterministic check passes or the deploy does not happen. No overrides, no just this once.
Level 2 is a tripwire. Trajectory drift on the golden traces beyond a small threshold blocks the deploy until a human looks at the diff and signs off. Sometimes drift is the improvement you intended; the point is that someone decides that on purpose.
Level 3 is a trend. The judge's pass rate must stay within the noise band of the baseline. A two-point dip is weather; a ten-point dip is a regression wearing a fluent disguise.

Then add the one ritual that keeps the suite alive: thirty minutes a week, one engineer and one person who knows the domain, reading ten sampled production traces. Every failure they find becomes a new test case. The suite grows out of production instead of imagination, which is the only reason it stays honest.

What the retrofit actually costs

On a typical engagement the arc looks like this. Day one: traces mined, first ten deterministic checks running in CI. End of week one: fifty golden traces snapshotted and replaying, labeled set done. End of week two: judge calibrated, deploy gate on, weekly trace review on the calendar. One engineer, part-time, alongside the normal roadmap.

The payoff usually arrives faster than teams expect. The first blocked bad deploy pays for the whole effort. The quieter win is what happens when a provider deprecates your model — with a suite, the migration is an afternoon of running evals against the replacement and reading the diffs; without one, it is a leap of faith with your customers underneath.

The short list, if you only do five things

Mine 50–100 real production requests and label them pass or fail with one line of why. This is the foundation everything else calibrates against.
Write ten deterministic checks for everything that has a right answer — schema, exact values, abstention, cost — and run them on every commit.
Snapshot ~50 golden traces and replay them on every change, asserting trajectory invariants rather than exact output strings.
Add an LLM judge only for what you cannot hard-code, with binary verdicts, calibrated to at least 85% agreement with your human labels, on a pinned model version.
Gate deploys on the suite and review ten production traces a week, folding every new failure back in as a test case.

None of this requires buying a platform, and none of it requires stopping feature work. It requires deciding that the phrase it looked fine in staging is no longer an acceptable release criterion — the rest is two weeks of plumbing.