Quality testing an AI agent is not the same problem as quality testing a web app. Assertion suites cover the deterministic parts. What they cannot cover is whether the agent asked the right next question, whether it stayed in scope, whether the read-back honestly reflected what the user said, or whether it sounded like a competent person doing the job. Those are judgment properties. String equality cannot see them.
Most teams try to build one testing framework that scores both the deterministic and the judgment surfaces. It does not work. Three months in, either the harness scores everything shallowly and produces numbers no one trusts, or the team gives up and ships on vibes. The failure mode is the same either way. Two different problems, one instrument.
Split the problem. Build the deterministic harness first. Design the probabilistic harness separately. The deterministic harness scores what has a right answer. The probabilistic harness calibrates itself against you for what does not.
I built the deterministic harness first, for a customer support triage agent. The probabilistic harness is the next thing I am designing. Here is how both work when you treat them as separate engineering problems.
The Deterministic Half
The customer support triage agent takes an inbound ticket and produces a structured record: category (billing, technical, account, general), priority (low, medium, high, urgent), a requires-human-review flag, and extracted entities (customer id, order id, product mentioned, sentiment). If the ticket is a reply to an existing thread, prior conversation state gets threaded through.
The deterministic half of the harness scores whether the agent parsed the ticket correctly. Nothing else. Not tone, not helpfulness, not whether the follow-up question was well chosen. Only whether the structured output matches what the input said.
The unit of evaluation is one extraction. A gold case is a triage input plus expected assertions over the output fields. Assertions use dotted paths: category, priority, requires_human_review, entities.customer_id, entities.order_id. Free-text fields (product mentioned, sentiment description) resolve with case-insensitive substring match. Everything else (categories, priorities, IDs, booleans) resolves with exact case-insensitive equality. A case passes when every assertion in it passes. The harness aggregates to a single percentage of passed assertions plus a cases-failed count.
That is the whole scoring model. There is no rubric. There is no LLM grading the output. String equality is the entire judgment.
The output is a number and an exit code. Zero if every case passed. One if any case failed. Two if configuration was broken. That exit code is what makes the harness usable in CI. A prompt change or a model change either raises the number or it does not.
Two other pieces make the harness reproducible.
A fixed clock injected into every run. If a ticket says “this happened yesterday,” the harness resolves “yesterday” the same way on every execution. Without this, half the gold cases become non-deterministic and the harness stops being a measurement instrument.
Every run stamps the model version and the prompt version that produced the score. A score without both stamps is a rumor. A score with both is a data point you can compare against.
The gold cases start small. Eight to twelve, drawn from actual customer messages the agent handled badly in production. Not a coverage matrix designed up front to be comprehensive. A regression net grown from observed failure. When a new failure appears, it becomes a new case in the harness. That is the growth rule. The harness gets bigger when reality forces it to.
Model comparison is trivial once this exists. Point the configuration at model A, run the harness, note the score. Point configuration at model B, run the harness, note the score. Whichever number is higher is the answer, unless the failing cases are ones you refuse to fail on. There is no swap process to build. The work went into the harness. The comparison is just running it twice.
Why This Is Not Enough
The deterministic harness answers one question well: did the agent parse the input correctly. It answers no other question at all.
It says nothing about whether the agent asked the right next question, whether it re-asked for something the customer already said, whether it stayed in scope when the customer asked for something the agent cannot do, whether the read-back at the end faithfully reflected what the customer described, or whether the tone was one a support agent should use.
Those are multi-turn judgment properties. They do not reduce to string equality. A ticket triage agent that scores 100% on extraction and re-asks the customer for their order number three times has passed the harness and failed the job.
You cannot quality test judgment with assertions. Which is where the second harness starts.
The Probabilistic Half
Quality testing judgment behavior has one design principle: engineer the objective parts as code, only send the genuinely subjective parts to a judge. The human-approval gate, the scope-guard refusals, the “never take an action the customer did not ask for” invariants are pass-fail properties and stay as code assertions. The judge grades only what actually requires taste.
The unit of evaluation shifts from one extraction to one whole conversation. Each scenario is a customer persona with a goal, a simulated user driver that plays that persona (either scripted turns or a second LLM playing the customer), a fixed clock so relative dates stay stable, and expected trajectory markers that name what the final state should look like and what invariants must hold.
The scenarios come from the same place the deterministic cases come from. Production conversations the agent handled badly. The persona is reverse-engineered from what the customer did.
Five dimensions get scored per scenario, chosen for the kind of agent this is.
Goal completion. Did the conversation reach a complete, valid, correct outcome the customer wanted?
Elicitation efficiency. Did the agent ask only for what was missing, handle out-of-order and partial information, and avoid redundant questions?
Groundedness. Are all field values traceable to what the customer said? Is the read-back an honest mirror or an invented one?
Safety and scope discipline. Did the agent stay in scope, refuse what it should not do, and never imply it took an action it did not?
Communication quality. Was the tone one a support agent should use?
Each dimension gets a 1 to 4 anchored scale with concrete descriptors. Not a vague 1 to 10 that averages to nothing. Even scale, no neutral middle, so the judge has to make a call. The judge returns per dimension a score, a one-sentence justification, and the specific turn indices in the transcript that support the score. That evidence-citation requirement is a quality control. A judge that must quote the turn it is grading hallucinates less than a judge that gets to generalize.
Aggregation is per-dimension means across the scenario suite. That is what you compare between prompt v5 and prompt v6, or between model A and model B. On top of the means sit hard gates: any scenario where a code invariant fails (agent took an unauthorized action, left scope, ignored a refusal case) is a hard fail regardless of how well the prose scored. Subjective scores never buy back a safety violation. A release bar is stated as thresholds. Ship only if every dimension mean is at or above 3.3, zero hard fails, and no single scenario scores a 1 on Groundedness.
What Makes the Judge Trustworthy
This is the part most published discussions of LLM-as-judge get wrong, and it decides whether the whole harness produces anything useful. A naive judge is worse than no evaluation at all, because it produces confident numbers that describe nothing.
Four design decisions make the difference.
Rubric-anchored, structured output. The judge gets the dimension definition, the four anchor descriptors verbatim, the full transcript, the scenario’s expected outcome, and returns structured JSON. No free-form “grade this.”
One dimension per judge call. Separate calls per dimension beat one call scoring all five. Less anchoring bias, sharper justifications, and different judge prompts can be tuned per dimension.
Different model as judge than as agent. If the agent runs on model X, the judge runs on model Y where possible, so the judge is not grading its own stylistic priors. At minimum, judge and agent are versioned independently.
Panel plus disagreement flag on the risky dimensions. For Groundedness and Safety, run three judges and take the majority. When they disagree, flag the scenario for human review rather than averaging. Disagreement is signal, not noise.
None of that matters without the one step you cannot skip: calibration against human labels. You hand-label twenty to thirty transcripts yourself. You measure judge-versus-human agreement per dimension. The judge is only trusted on dimensions where it agrees with you. Where it does not, you rewrite the anchors and re-calibrate. Without this step, the harness produces confident numbers that mean nothing. Judge scores are reported as trends across a suite, never as single authoritative numbers. That is the opposite of the deterministic harness, which can claim an exact pass count. Both instruments produce numbers. Only one of them produces truth.
How the Two Halves Fit Together
The two harnesses answer different quality questions, but they work the same way underneath. Gold cases seeded from production conversations, grown when reality forces it. Fixed clock so time-sensitive inputs stay deterministic. Every score stamped with model version, prompt version, and (for the probabilistic half) judge version and scenario suite version. Config-driven A and B by re-pointing configuration and running the suite twice.
What differs is what they measure.
The deterministic harness measures whether the agent parsed correctly. Truth is string equality. Output is an exact percentage and an exit code.
The probabilistic harness measures whether the agent behaved like a competent support person. Truth is calibrated against human judgment. Output is per-dimension means, hard fail counts, and flagged disagreements for human review.
The scope-split lets each harness be honest about what it is. Neither one tries to score the other one’s half.
Triage, Not Oracle
The final position on the probabilistic harness matters more than any of the mechanics. It is a triage and trend instrument, not an oracle. Its job is to catch regressions and surface transcripts for human eyes cheaply. Not to replace human judgment. The moment you treat its numbers as ground truth instead of a compressed proxy for human review, it stops being trustworthy.
The deterministic harness can claim to know things. String equality is not a matter of opinion. The probabilistic harness cannot claim to know anything the humans who calibrated it did not already know. It just applies their judgment at scale, cheaply, so they can spend attention on the transcripts that matter.
Any published discussion of AI agent evaluation that skips this framing produces harnesses that do not survive contact with production. The number goes up, the release ships, the customer conversations get worse, and no one knows why until they read a transcript.
Score what has a right answer. Calibrate against yourself for what does not. Grow both harnesses when reality forces you to. Refuse to treat any of the numbers as truth they are not.
That is the whole approach to quality testing an agent that thinks.