Back to blog
Guides

An agent evaluation framework you can trust

Frozen answers and held-out sets: four rules for knowing whether a fix to your AI agent worked, shown on real evaluation runs from a support agent.

On this page

We ran the same 16 test cases against three versions of a support agent: a healthy one, one with a bug, and one with the bug fixed.

The fix worked. It brought one score back up by 6.3 points. But on a different score, the buggy version came out 6.3 points ahead of the healthy one. Nothing about it was better. One test case simply passed in that run and failed in the next.

A real fix and a lucky run moved the numbers by the same amount. From the scores alone, you could not tell which was which.

That is the problem with checking a fix by rerunning the cases that failed. A case can pass after a fix for four different reasons, and only one of them is that your agent got better.

This post is about the other three reasons, and about four rules that remove them. The examples come from the public demo project in traceroot, a support agent for an outdoor gear shop with a 16-case golden set. Every number in this post is from that project.

A before-and-after comparison you can trust

One pool of cases with expected answers is split into a dev set and a held-out set. One before run scores both sets to get a baseline. Only the dev set goes through the fix loop. Both sets are scored again in an after run, and the held-out set is scored once.

Both sets are scored before and after. Only the dev set goes through the fix loop.

What is an agent evaluation framework?

An agent evaluation framework is the set of parts and rules you use to decide whether a change to an AI agent made it better. The parts are familiar: a dataset of cases, each with an input and an expected answer; a task that runs your agent on a case; scorers that grade each result; and runs that record what happened.

Most writing about agent evaluation stops at the parts. Which metrics to track, how to trace tool calls, where to put a judge model. Those matter, and we covered the metrics side in The 6 LLM evaluation metrics that actually matter for AI agents in production.

The rules get less attention. Which cases are you allowed to look at? What has to stay fixed between two runs? How many times may you score the same set? A framework without those rules still produces numbers. It just cannot tell you which of them to believe.

Why rerunning the failures overstates the fix

Saving a production failure as a test case is the right first move. We walked through it in Testing a LangGraph SQL agent fix with a regression eval. A saved case answers one question well: does this known failure come back? It cannot answer a second question: does the fix work on cases you have not seen?

Here are the three reasons a fix can look better than it is.

Reason 1: You selected the cases because they failed. Agents are not deterministic, so some cases pass on one run and fail on the next with no change at all. When you pick out the failures and rerun only those, some of them will pass the second time whether you fixed anything or not, and the fix gets the credit. Statisticians call this regression to the mean: unusual measurements tend to be followed by ordinary ones, and the effect is strongest when you follow up only on a group chosen for its bad first result (Barnett et al., 2005).

The demo project shows it. Here are three runs of the same 16 cases. Blue is the agent before anything went wrong. Purple is the same agent after its refund limit was changed by mistake, from $200 to $2000, so a $419 refund for a torn sleeping bag no longer went to a supervisor. Orange is the agent after the fix.

Run comparison in traceroot with three runs of the support agent golden set; red boxes mark the case the fix repaired and a case that flipped by chance

Top box: the case the fix repaired. Bottom box: a case that flipped by chance. Blue, purple and orange are the healthy, buggy and fixed runs.

The bug does what you would expect: it breaks the refund case. The $419 refund drops from 100% to 0% on expected_facts, and the fix brings it back. Now look at required_tools. One case, a customer asking for a manager about a late delivery, fails in the healthy version, passes in the buggy one, and fails again after the fix. That case never reaches the refund limit. Nothing changed for it, yet it flipped twice. An agent can hear the same question and give a different answer.

With 16 cases, one flip is 6.3 points. The fix moved expected_facts by exactly that much, and chance moved another metric by the same amount in a version that was broken. If the late-delivery case had been the failure you were chasing, rerunning it with no change at all would have shown it fixed.

Reason 2: You tuned the fix to those cases. Each time you read a failing transcript and edit the prompt, a little of that transcript ends up in the prompt. After enough rounds the agent handles those cases well and you have learned nothing about the next one. This is overfitting, and it does not need bad intent. Research on agent benchmarks found the same pattern at scale: many benchmarks have weak holdout sets or none, and agents end up fitted to the benchmark (Kapoor et al., 2024).

Reason 3: The yardstick moved. Between the two runs someone loosened an expected answer, added three cases, or reworded the judge prompt. The score went up. So did the ruler.

The four rules below take these three reasons apart one at a time.

Rule 1: Freeze the answers before the first run

Frozen answers means you decide what "correct" is before you see any output, and you do not change it until the comparison is over. The before run and the after run have to use the same cases, the same expected answers and the same scorers. Otherwise a higher score may only mean an easier test. The yardstick has two halves.

The expected answers. Write them down first. For the sleeping bag case, the demo expects the answer to mention a supervisor or a review. If that expectation were written after reading the agent's reply, it would describe what the agent did, not what it should do.

The scorers. How you grade is part of the yardstick too. The demo uses three scorers written in code and one LLM judge, a second model that checks each answer against the shop's policies. Change that judge's prompt or model between the before run and the after run, and you can no longer say how much of the difference came from the agent.

Calibrating a judge is separate work that happens earlier, against human labels, with its own dev and test split. Hamel Husain's evals FAQ describes that process. Once the judge is calibrated, it is frozen like everything else.

Freezing only helps if you can check it afterwards. In traceroot, every published dataset version is immutable. Editing a case never rewrites history; it publishes a new version. The demo's golden set has two: the first with 12 cases, the second with 16.

Dataset page in traceroot with the version menu open and outlined, listing two versions of the support agent golden set

Boxed: both versions of the golden set.

Each evaluation run records the version it measured.

Evaluation runs list in traceroot; red boxes mark two runs with the same run name and commit but different dataset version ids

Left: same code. Right: different dataset versions.

Runs #1 and #2, the bottom two rows, tested exactly the same code. Run #1 used the first version of the golden set, with 12 cases. Run #2 used the second, with 16. Any gap between their scores could come from the four added cases instead of the agent. The list makes that visible before anyone compares them. Runs #2 through #6 all used the second version. That is why the comparison earlier in this post is a fair one.

So when you rerun after the fix, ask for the same dataset version the before run used. Do not just take the latest one:

python
import subprocess
from traceroot import evaluate, pull_dataset_version

BEFORE_VERSION = "..."  # dataset_version_id recorded by the before run
commit = subprocess.run(
    ["git", "rev-parse", "--short", "HEAD"], capture_output=True, text=True
).stdout.strip()

frozen = pull_dataset_version(BEFORE_VERSION)

result = evaluate(
    name="Support agent golden set",
    evaluation_key="support-golden",
    dataset=frozen,
    task=fixed_agent,
    scorers=SCORERS,              # the same list the before run used
    candidate_version=f"git:{commit}",
)

Keep the scorers in the same repository as the agent, and have the eval script read the current commit and pass it as candidate_version, as the example does. Later, when you want to know how a score was graded, that commit id takes you to the exact judge prompt that was used. The Python SDK also attaches the git repository and commit to each run's metadata, so the record exists even if you forget the label.

Rule 2: Hold out cases before you start fixing

Start with one pool of cases, each with an expected answer. Before you read a single failure, split it into two sets: a dev set you tune on and a held-out set you leave alone. Both come from the same pool. The held-out set exists to answer one question at the end: did the fix help on cases that had no say in it?

Three details decide whether a holdout works.

Split by scenario, not at random. A pool of agent cases is full of near-duplicates. The demo set has three questions that all turn on the return policy: worn shoes, a jacket that does not fit, and a return bundled with an order status question. Put siblings on both sides and the holdout already knows the answer. Group cases by the scenario or failure type they came from, and assign whole groups to one side.

Give the two sets different rules. Anyone can read dev failures, all day. Nobody reads individual holdout failures while the fix is in progress. You may know the holdout score. You may not open the failing transcripts and patch for them. Do that and the holdout becomes a second dev set.

Score it once. The model never sees the held-out answers either way. The leak is you. Every decision you make after looking at holdout results carries a little of that set into the agent.

In traceroot, the simplest way to keep the two sets apart is two datasets, each with its own key:

python
from traceroot import Dataset

dev = Dataset("Support agent dev", key="support-dev")
holdout = Dataset("Support agent holdout", key="support-holdout")

for case in cases:
    target = holdout if case.scenario in HOLDOUT_SCENARIOS else dev
    target.add(case.input, expected=case.expected, metadata=case.metadata)

Run the dev dataset as often as you like. Run the holdout twice per fix: once before you start, once when you are done.

Rule 3: Rerun the unfixed version too

The cleanest test for luck is to do nothing and measure again. Before you credit a fix, rerun the old version on the same cases. However many failures pass with no change, that is the amount your fix has to beat.

The same idea shapes how you read two runs. Do not compare averages. Compare case by case, and count the flips in both directions: how many cases went from fail to pass, and how many went from pass to fail. Anthropic's guidance on the statistics of evals makes the same point about paired differences. A result of "four fixed, three broken" and a result of "one fixed, none broken" can produce the same change in the average, and they are very different fixes.

Small sets make this matter more. On 16 cases every flip is worth 6.3 points, so a single unlucky case hides a real fix or invents one. On 40 cases a flip is 2.5 points. If you cannot add cases, run each one several times and look at how often it passes.

Rule 4: Retire a used holdout and refill it

A holdout wears out. If the score is bad and you go back to fix more, that set has now shaped your decisions. It still has a job, just a different one.

  • Move it to a regression set. These cases must keep passing forever. They no longer count toward the number you report, because they are no longer unseen.
  • Refill from new production traces. Fresh traffic is the one source of cases nobody on the team has thought about. In traceroot, Add to datasets saves a span or a trace as a case, with the expected answer you write for it.
  • Do not refill only from failures. A holdout built from the latest failures is selected the same way your dev set was. Take ordinary traces too.

A bad holdout score is not a wasted run. It is the result you built the holdout to get, and it arrived before your users found it.

The four rules at a glance

Rule What it protects against How to check it
Freeze the answers Expected answers or judge edited after seeing results Both runs record the same dataset version and use the same scorers
Hold out cases A fix fitted to the cases you were reading Holdout score reported next to the dev score
Rerun the unfixed version Failures that pass by themselves Flips counted in both directions, against a no-change rerun
Retire and refill A holdout that has turned into a dev set Used cases moved to a regression set, new ones from fresh traces

What a clean result still leaves open

A clean held-out result says the fix carries over to cases like the ones you sampled. It says nothing about scenarios missing from both sets, and nothing about what happens when traffic shifts.

That is why offline evaluation is half of the loop. In traceroot, detectors keep judging production traces after you ship, and Signals group the traces that share a problem. A resolved Signal reopens by itself when a new trace shows the same issue. That is the production-side answer to the question your holdout approximates.

Try it on your own agent

traceroot is open source and self-hostable. Datasets, immutable dataset versions, evaluation runs and the run comparison in this post are all in the current release, and the evaluation docs cover the SDK calls in Python and TypeScript. The demo agent is public too, at traceroot-ai/demo-support-agent, with the golden set and the fault branches used above.

Start small: pick the next fix on your list, publish the cases before you touch the agent, and set a third of them aside.

FAQ

What is an agent evaluation framework?

It is the combination of a dataset of cases, a task that runs your agent, scorers that grade the results, and rules for when two runs can be compared. The rules are what make a before-and-after result trustworthy: the expected answers and scorers stay fixed, and some cases are kept out of the fixing process.

What is a held-out set in agent evaluation?

A held-out set is a group of cases that nobody uses while changing the agent. You do not read its failures and you do not tune on it. You score it once the fix is finished, so it measures how the fix performs on cases that did not influence it.

What happens if the held-out score is bad?

Then the fix did not carry over, which is what you needed to know. You can keep working, but that held-out set has now informed your decisions. Move it to a regression set and draw a new holdout from recent production traces before you report another number.

How many cases does a held-out set need?

Enough that one case is not the whole story. With 16 cases a single flip moves the score by 6.3 points. With 40 it moves by 2.5. If you have few cases, run each one several times and compare case by case.

Does freezing the dataset mean it can never change?

No. Datasets should grow as production shows you new scenarios. The rule is narrower: within one comparison, the before run and the after run use the same dataset version and the same scorers. New cases go into the next comparison.