How to evaluate an agent before trusting it with operations

The model provider announces a new version, faster and cheaper. The team switches it on on a Friday afternoon, because the change looks minor. On Monday, the expense classification agent is still working, but it has started sending one type of invoice it used to handle well to the wrong category. Nobody notices until month-end close.

The scene is a composite and describes a regression: a change that fixes one thing and breaks another. Automated tests help detect software regressions. Agents also need evaluations, or “evals”, of answer quality, tool selection and escalation to a person.

Why the demo deceives

A demo shows the case the agent handles well. Real operations are made of edge cases: the invoice in two currencies, the payment that bundles three documents, the customer whose name is written four ways. An agent can get the vast majority of easy cases right and fail precisely on the ones that matter most. The only way to know is to test it against the real cases.

The reference set

The core of an evaluation is a set of reference cases: process examples with expected outcomes or criteria defined by the people who know the business. A reconciliation may have one correct match; an open-ended answer may allow several formulations and need a rubric for accuracy, evidence and limits. Keep a representative sample to estimate expected performance and a separate stress set with more rare, costly cases. If real data is used, protect personal information and keep test cases out of instructions and training.

An agent that isn't measured against real cases hasn't been tested. It has been presented.

What gets measured

• Accuracy: for closed cases, the share matching the correct outcome; for open-ended cases, how well answers meet the defined rubric.
• Coverage: what share of cases the agent resolves, and what share it hands to a person.
• Quality of hand-off: when it isn't sure, does it say so, or answer confidently anyway?
• Cost and time per case, to know whether the improvement pays off.
• Safety: whether it attempts actions outside its scope or exposes data it shouldn't.

Of all of them, the third is the most often forgotten and the one that matters most. An agent that can say “I'm not sure, please review” is worth more than one that's slightly more accurate but never doubts.

Evaluation isn't an event: it's a loop

Evaluation isn't done once before launch. It's repeated whenever something changes: a new model version, an adjustment to the instructions, a new tool or a modified business rule. If performance worsens on a critical type of case, review the change before production. The reference set grows with operations: detected errors become test cases, though no set can guarantee they won't recur.

It looks like a lot of work, and at first it is. But it's the same work that lets you raise an agent's autonomy with arguments, as we saw in the previous episode, rather than by intuition.

The scenes in these episodes are composites of common situations, not customer cases. Any figures in them are illustrative.