Most teams build an agent first and ask how to measure it later. By then, the question has become political: the demo impressed people, and nobody wants a number that might disagree with it.
We reverse the order. Before writing a prompt, we write down what a correct outcome looks like for fifty to two hundred real cases, taken from the work the system is meant to do. That set becomes the first artefact of the project and the last thing we check before every release.
What goes into the set
- Real inputs, not invented ones. Historic tickets, documents, requests: the messier, the better.
- An expected outcome per case, written by the person who would otherwise do the work.
- Cases the system should refuse, escalate or hand back. These matter more than the easy ones.
What we measure
We avoid one aggregate score. A single number hides the failure modes that matter. Instead, each suite reports a handful of properties separately: task success, faithfulness to sources, policy compliance, cost and latency. A change that improves success while degrading policy compliance is a regression, not progress.
A model upgrade is a change like any other. It goes through the same suite as a prompt edit.
Why this changes the architecture
Once evaluation runs in CI, the system has to be built so it can run there. Tools need deterministic fakes. Retrieval needs fixed snapshots. Model calls need to be routed through one gateway so they can be replayed. These constraints make the production system better too: more modular, more observable, easier to change.
The suite also makes provider decisions cheap. When a new model arrives, the question "is it better for us?" takes an afternoon rather than a quarter.
Where to start
Pick the narrowest valuable task. Collect the cases. Agree on what "correct" means with the person who owns the outcome. Only then decide how much autonomy the system deserves.