Most AI features we are asked to review have the same shape. There is a prompt, it is long, it has been edited perhaps forty times, and nobody can say whether edit forty was better than edit twelve. Someone tried a change, it looked good on the two examples they had open, and it shipped.
That is not a prompt problem. That is a missing measurement problem, and it is the one thing that separates an AI feature you can improve from one you can only fiddle with.
Why "it looks good" fails
Language models fail unevenly. A change that improves the common case often degrades an uncommon one, and the uncommon one is usually where the money or the liability is. Without a fixed set of cases you re-run every time, you cannot see the trade — you only see the examples you happened to look at, which are the examples that prompted the change.
The fix is unglamorous: a regression set. Ours for AI Trainer sits at 1,240 cases. It grew one case at a time, mostly from real failures.
What is actually in it
Four buckets, and the proportions matter more than the total.
Golden answers (about 30%). Questions with a single defensible answer, drawn from the source documents. These catch outright regressions.
Adversarial and edge cases (about 35%). Questions where the documents disagree with each other, where the answer changed between policy versions, where the question contains a false premise. This is the largest bucket on purpose, because it is where quality actually lives.
Should-refuse cases (about 20%). Questions the system must decline: outside the corpus, asking for medical or legal advice, asking it to commit to a price it cannot know. A model that answers these confidently is worse than one that answers nothing.
Format and grounding checks (about 15%). Is the citation attached? Does the citation actually contain the claim? Is the output parseable by whatever consumes it?
The four metrics we publish, and what each is for
Faithfulness — does the answer follow from the retrieved context, rather than from the model's general knowledge? This catches the plausible-sounding invention, which is the failure mode that costs trust fastest.
Answer recall — of the facts a complete answer needs, how many appear? Faithfulness without recall gives you an answer that is true and useless.
Citation match — does the cited passage actually support the sentence it is attached to? Citations that look right and point at the wrong paragraph are worse than no citations, because they buy trust they have not earned.
Refusal rate — measured against the should-refuse bucket only. We expect this to be high there and near zero elsewhere. A single global refusal number tells you nothing.
What we deliberately do not measure
Model-graded "helpfulness" as a headline number. We use an LLM judge for triage, because a human cannot read 1,240 outputs per run. We do not report it as a quality score. Judges drift, they favour longer answers, and they are correlated with the thing being tested when both are the same model family.
Aggregate scores across buckets. One number that averages golden answers with should-refuse cases can improve while both halves get worse. We report per-bucket or not at all.
Latency in the same table as quality. They trade against each other and mixing them encourages quietly buying quality with a response time the product cannot afford. Separate budgets, separate conversations.
Running it
The harness runs on every prompt change, every retrieval change and every model version bump. It takes about eleven minutes. A change that drops any bucket by more than two points does not ship without someone writing down why.
The first version of this took a day to build and was worth it within a week. It found that a retrieval change we had been about to celebrate improved recall by four points and cost nine points of citation match, because it was pulling in longer passages that happened to contain the answer somewhere.
That is the entire argument. Not that evaluation is virtuous, but that without it you cannot tell an improvement from a trade, and you will keep making trades you did not intend.