The first eval set for an agent is usually written by hand. Someone sits down, writes thirty prompts they think are representative, writes the expected answers, and wires up a score. It is a reasonable start. It is also a set of guesses about what users will do, written by the people least likely to be surprised by the agent.
Then the agent goes to production and users do something else. The failures that matter are almost never the ones in the hand-written set.
The fix I keep coming back to is simple to say: your traces are your eval dataset. If you already record what the agent did in production, you already have the raw material for the evals that matter most. You just need a path from "this went wrong" to "this can never silently go wrong again".
The problem: two systems that never meet
In most setups I have seen, observability and evals are owned by different people and live in different places.
- Observability answers "what happened?" Traces, spans, token counts, latency, tool calls, errors. It is built for debugging one bad request at a time.
- Evals answer "is this version good enough to ship?" A dataset, a grader and a score. They run in CI or in a notebook before a release.
When the two never meet, the same pattern repeats:
- A user reports a bad answer. Someone finds the trace, understands the bug, changes the prompt, and ships.
- Nobody adds that case to the eval set, because the eval set lives somewhere else and has its own format.
- Three weeks later a different prompt change brings the bug back. The evals stay green because they never knew about it.
This is the same thing that happens with a production incident that never gets a regression test. In normal software we learned to close that loop. With agents, a lot of teams have not yet.
The loop
The loop I want looks like this:
production ──► traces ──► triage ──► eval case ──► CI gate ──► release
▲ │
└──────────────────────────────────────────────────────────┘
- Every agent run produces a trace with enough detail to replay it.
- Failures get flagged: by a user, by a cheap automatic check, or by sampling.
- A person looks at the flagged trace and decides what "correct" would have been.
- The trace becomes an eval case, with the inputs frozen and the expectation written down.
- CI runs the eval set on every change to the prompt, model, tools or code.
- The next release ships with that failure covered.
Nothing in this loop is new. It is incident, postmortem, regression test, applied to a system whose behaviour is not deterministic.
A worked example: an ops triage agent
Let me make this concrete with the kind of agent I care about: one that gets paged alongside the on-call engineer. It reads the alert, queries metrics and logs through MCP tools, looks at recent deploys, and proposes a likely cause with the evidence behind it.
A typical run looks like this in a trace:
agent.run alert=HighErrorRate service=checkout ns=prod
├── llm.call model=… input_tokens=3.1k
├── tool.call prometheus.query rate(http_requests_total{status=~"5.."}[5m])
├── tool.call k8s.list_events namespace=prod
├── tool.call deploys.recent service=checkout
├── llm.call model=… output_tokens=420
└── agent.result cause="bad deploy 14:02, new DB pool size" confidence=0.7
Now suppose an on-call engineer marks one run as wrong. The agent blamed a deploy, but the real cause was a node that ran out of disk. When you open the trace, the agent queried k8s.list_events in namespace=staging instead of prod, saw nothing interesting, and fell back to the most recent deploy.
That is a great eval case. It is real, it is specific, and it describes a failure mode (wrong namespace, then a plausible but wrong conclusion) that a hand-written set would probably never include.
What a trace needs to carry
You can only replay what you recorded. For a trace to become an eval case, I want each run to record at least:
- The full input. The alert payload, the system prompt version, and any retrieved context. Not a summary.
- Every tool call with arguments and results. The results matter most: they are what you will replay in place of live systems.
- Model and config. Model ID, temperature, and the prompt or agent version.
- The final output. What the agent concluded and what it did.
- A run ID that feedback can attach to. Thumbs down, "wrong cause" or a correction from the engineer all need to point back at one trace.
If you use OpenTelemetry, the GenAI semantic conventions (gen_ai.* attributes) cover the model side, and tool calls fit naturally as child spans. The specific backend matters less than this rule: tool results are part of the trace, not just tool names. Without them you cannot replay the run.
Two cautions. Tool results from production can contain secrets and customer data, so scrub them at write time, not later. And sample with intent: keep every run that got negative feedback or errored, and a small random slice of the rest.
Turning a trace into an eval case
The key decision is what to freeze and what to re-run. For an agent, I freeze the world and re-run the agent:
- The alert and the context are fixed.
- Tool calls are answered from the recorded results, not from live Prometheus or Kubernetes.
- The model, prompt and agent code are whatever version is under test.
That makes the eval repeatable. Production state moves on, but the case does not.
A case on disk might look like this:
# evals/triage/cases/2026-10-checkout-disk-pressure.yaml
id: checkout-disk-pressure
source_trace: 7f3c9e1a…
input:
alert: { name: HighErrorRate, service: checkout, namespace: prod }
tool_fixtures:
- match: { tool: k8s.list_events, args: { namespace: prod } }
result: fixtures/prod-events-diskpressure.json
- match: { tool: k8s.list_events, args: { namespace: staging } }
result: fixtures/staging-events-empty.json
- match: { tool: deploys.recent, args: { service: checkout } }
result: fixtures/checkout-deploys.json
expect:
cause_contains_any: ["disk pressure", "DiskPressure", "out of disk"]
must_call:
- { tool: k8s.list_events, args: { namespace: prod } }
must_not_call:
- { tool: k8s.delete_pod }
Note what I recorded on purpose: the staging fixture returns nothing, just like in production. If the agent makes the same mistake again, it will see the same empty result and probably reach the same wrong answer. The case reproduces the failure, not only the input.
The conversion itself is a small script: read the trace, pull out the input and tool results, write fixtures, and leave expect for a person to fill in. I do not let the script guess the expectation. Deciding what "correct" means is the valuable part, and it needs someone who understands the incident.
Grading: check the path, not only the answer
For an agent like this, the final text is only half the story. A run can land on the right cause for the wrong reason, or the wrong cause after doing everything right. I grade at three levels:
- Outcome. Did the proposed cause match? For causes I prefer simple checks (
contains_any, a label match) over a model judge where possible. They are cheap, stable and easy to argue about. - Trajectory. Did it call the tools it had to call, with the right arguments? Did it avoid tools it must never call? The wrong-namespace bug is a trajectory failure, and a trajectory check catches it even when the agent gets lucky on the answer.
- Safety. Hard failures, scored separately: any destructive call, any action outside the alert's namespace, any secret in the output. One of these fails the case no matter what else happened.
A grader for the trajectory part does not need to be clever:
def grade_trajectory(case, run):
calls = [(c.tool, c.args) for c in run.tool_calls]
missing = [m for m in case.expect.must_call if not any(matches(m, c) for c in calls)]
forbidden = [c for c in calls if any(matches(f, c) for f in case.expect.must_not_call)]
return {
"pass": not missing and not forbidden,
"missing": missing,
"forbidden": forbidden,
}
When I do need a model judge, for example to decide whether the evidence actually supports the cause, I treat it like any other dependency: pinned model version, a written rubric, and a small set of human-labelled cases to check it still agrees with people.
Running it in CI
Once cases live in the repo, the CI part is familiar:
- Run the full triage eval set on any change to prompts, tool definitions, the model ID or agent code.
- Run each case several times. One run per case is a coin flip for anything non-deterministic. I use three to five runs and report the pass rate per case.
- Gate on two things: no regression on cases that came from real incidents, and no safety failure, ever. The overall score can wobble a little; those two cannot.
- Show the diff in the pull request: which cases flipped from pass to fail, with a link to the run trace. Reviewers should see "checkout-disk-pressure: 5/5 → 2/5" without digging through logs.
The evals also produce traces, in the same format as production. When a case fails in CI, you debug it with the same tools you use for a production incident. That symmetry is a large part of why this approach works.
Where the cases come from
After a few months, the eval set looks very different from the hand-written one it started as. The sources I lean on, in order of value:
- Negative feedback from users. Every "this was wrong" with a trace attached is a candidate case.
- Incidents and near-misses. Anything the agent touched during a real incident gets reviewed, even if it was right. Correct runs during hard incidents make strong positive cases.
- Automatic flags. Runs that hit tool errors, timeouts, very long trajectories or low confidence. Cheap to detect, often interesting.
- Random samples. A small weekly sample keeps the set honest about ordinary traffic, not only the failures.
- Hand-written cases. Still useful for things that have not happened yet, especially safety cases. But they become the minority.
I also delete cases. When a tool is removed or an alert type no longer exists, cases built on it go too. An eval set nobody maintains turns into noise, just like a flaky test suite.
Tradeoffs
This approach is not free.
- Storage and privacy. Recording full tool results costs storage and pulls production data into a dev workflow. Scrub at write time, set retention, and keep fixtures in a place with the right access controls.
- Fixtures go stale. A recorded Prometheus response from six months ago may not match today's metric names. When tool schemas change, cases need updating, the same way mocks do.
- Frozen worlds hide some bugs. Replaying recorded results means the agent cannot explore a path it did not take in production. If it now calls a tool with arguments nobody recorded, the case has to fail loudly or fall back to a clear "no fixture" result. Never call the live system.
- Human time. Writing the expectation for each case takes a person who understands the incident. That is the expensive step, and the one I would not automate away.
- Overfitting. A set built only from past failures can make you very good at last quarter's bugs. Keep the random samples and the hand-written safety cases in the mix.
Checklist
- Every agent run produces a trace with full inputs, tool arguments, tool results, model and prompt version.
- Feedback (user, on-call, automatic) attaches to a run ID.
- Sensitive data is scrubbed when the trace is written.
- A script turns a trace into a case with fixtures; a person writes the expectation.
- Cases replay tool results from fixtures and never call live systems.
- Grading covers outcome, trajectory and safety separately.
- CI runs each case several times and gates on incident regressions and safety failures.
- Eval runs emit traces in the same format as production.
- The set is pruned when tools or alerts change.
Hand-written evals tell you whether the agent does what you expected. Evals built from traces tell you whether it still gets wrong what it already got wrong once. The second set is the one I would block a release on.