Evals before launch: how we test an AI agent

Every agent looks good in its first demo. The person showing it picks the request, knows the data and stops before anything goes wrong. Production is different: requests arrive with typos, missing details and edge cases nobody thought of, and the agent has to handle all of them.

Evaluation is how we close that gap. We treat it as part of the build from the first week, with its own test set, its own scoring and its own place in the release process.

Build the test set from real requests

The best evaluation set comes from the work the agent will do. For a support agent we pull real tickets. For an internal knowledge assistant we collect the questions staff actually ask, from chat logs, email threads and help-desk queues.

We clean out personal data, then sort the requests into groups:

• Routine cases that make up most of the volume and should simply work
• Hard cases that need several steps, several systems or a judgment call
• Edge cases with missing information, conflicting data or unusual wording
• Out-of-scope requests the agent should decline or hand to a person
• Adversarial inputs that try to make the agent ignore its instructions or reveal data

For each request a domain expert from the client team writes what a good outcome looks like. That expert's judgment is the standard the agent is measured against.

Grade the whole run

An agent does more than produce text. It reads data, calls tools and changes records, so we grade the full run:

• Outcome: did the task end in the right state? Was the order changed correctly, the ticket routed to the right queue, the refund declined when policy says so?
• Path: did the agent call the right tools with the right arguments, and avoid calls it had no reason to make?
• Answer: is the reply accurate, grounded in the source data and written in the client's voice?
• Hand-off: when the agent should have stopped and passed the case to a person, did it?

Outcome and path checks are mostly automatic, because we can inspect the system state and the tool calls. Answer quality uses a mix of automatic graders and human review. We calibrate automatic graders against human scores on a sample before we trust them, and we keep checking them as the agent changes.

Make it part of every change

The evaluation set runs on every change: a new prompt, a new model version, a new tool or a change to retrieval. Results are compared with the previous run, and any drop in a group blocks the release until someone looks at it.

This matters most when models change. A new model version can improve the average and still break a specific group of cases the business cares about. With a fixed test set we see that before customers do.

Decide what ready means

We agree on release thresholds with the client before testing starts. Routine cases need to pass at a high rate. Out-of-scope and adversarial cases need to be handled safely every time, because a single bad answer there can do real damage. Hard cases can launch at a lower rate if the agent hands off reliably when it is unsure.

Launch is usually staged. The agent first runs in shadow mode, drafting answers that a person reviews and sends. Then it handles the routine group on its own while hard cases still go to people. Each step widens only when the numbers from the previous step hold.

Keep the set growing

The test set at launch is a starting point. Every failure in production becomes a new test case, reviewed and added to the set. After a few months the set reflects what customers actually send, and it becomes one of the most valuable assets of the project.

What we recommend

If you are building an agent, start the evaluation set before the first prompt. Collect real requests, have an expert define good outcomes, and grade the whole run. It is slower in week one and much faster every week after, because you always know whether the last change made things better or worse.