← All blogs

AI ·

Your Traces Are Your Eval Dataset

How to turn agent traces from production into regression evals, so every failure an agent makes in the real world becomes a test it has to pass before the next release.

Manoj Paramsetti

Manoj Paramsetti

DevOps Engineer at SurveySparrow

The first eval set for an agent is usually written by hand. Someone sits down, writes thirty prompts they think are representative, writes the expected answers, and wires up a score. It is a reasonable start. It is also a set of guesses about what users will do, written by the people least likely to be surprised by the agent.

Then the agent goes to production and users do something else. The failures that matter are almost never the ones in the hand-written set.

The fix I keep coming back to is simple to say: your traces are your eval dataset. If you already record what the agent did in production, you already have the raw material for the evals that matter most. You just need a path from "this went wrong" to "this can never silently go wrong again".

The problem: two systems that never meet

In most setups I have seen, observability and evals are owned by different people and live in different places.

When the two never meet, the same pattern repeats:

This is the same thing that happens with a production incident that never gets a regression test. In normal software we learned to close that loop. With agents, a lot of teams have not yet.

The loop

The loop I want looks like this:

production ──► traces ──► triage ──► eval case ──► CI gate ──► release
     ▲                                                          │
     └──────────────────────────────────────────────────────────┘
  1. Every agent run produces a trace with enough detail to replay it.
  2. Failures get flagged: by a user, by a cheap automatic check, or by sampling.
  3. A person looks at the flagged trace and decides what "correct" would have been.
  4. The trace becomes an eval case, with the inputs frozen and the expectation written down.
  5. CI runs the eval set on every change to the prompt, model, tools or code.
  6. The next release ships with that failure covered.

Nothing in this loop is new. It is incident, postmortem, regression test, applied to a system whose behaviour is not deterministic.

A worked example: an ops triage agent

Let me make this concrete with the kind of agent I care about: one that gets paged alongside the on-call engineer. It reads the alert, queries metrics and logs through MCP tools, looks at recent deploys, and proposes a likely cause with the evidence behind it.

A typical run looks like this in a trace:

agent.run  alert=HighErrorRate service=checkout ns=prod
├── llm.call            model=…  input_tokens=3.1k
├── tool.call           prometheus.query   rate(http_requests_total{status=~"5.."}[5m])
├── tool.call           k8s.list_events    namespace=prod
├── tool.call           deploys.recent     service=checkout
├── llm.call            model=…  output_tokens=420
└── agent.result        cause="bad deploy 14:02, new DB pool size" confidence=0.7

Now suppose an on-call engineer marks one run as wrong. The agent blamed a deploy, but the real cause was a node that ran out of disk. When you open the trace, the agent queried k8s.list_events in namespace=staging instead of prod, saw nothing interesting, and fell back to the most recent deploy.

That is a great eval case. It is real, it is specific, and it describes a failure mode (wrong namespace, then a plausible but wrong conclusion) that a hand-written set would probably never include.

What a trace needs to carry

You can only replay what you recorded. For a trace to become an eval case, I want each run to record at least:

If you use OpenTelemetry, the GenAI semantic conventions (gen_ai.* attributes) cover the model side, and tool calls fit naturally as child spans. The specific backend matters less than this rule: tool results are part of the trace, not just tool names. Without them you cannot replay the run.

Two cautions. Tool results from production can contain secrets and customer data, so scrub them at write time, not later. And sample with intent: keep every run that got negative feedback or errored, and a small random slice of the rest.

Turning a trace into an eval case

The key decision is what to freeze and what to re-run. For an agent, I freeze the world and re-run the agent:

That makes the eval repeatable. Production state moves on, but the case does not.

A case on disk might look like this:

# evals/triage/cases/2026-10-checkout-disk-pressure.yaml
id: checkout-disk-pressure
source_trace: 7f3c9e1a…
input:
  alert: { name: HighErrorRate, service: checkout, namespace: prod }
tool_fixtures:
  - match: { tool: k8s.list_events, args: { namespace: prod } }
    result: fixtures/prod-events-diskpressure.json
  - match: { tool: k8s.list_events, args: { namespace: staging } }
    result: fixtures/staging-events-empty.json
  - match: { tool: deploys.recent, args: { service: checkout } }
    result: fixtures/checkout-deploys.json
expect:
  cause_contains_any: ["disk pressure", "DiskPressure", "out of disk"]
  must_call:
    - { tool: k8s.list_events, args: { namespace: prod } }
  must_not_call:
    - { tool: k8s.delete_pod }

Note what I recorded on purpose: the staging fixture returns nothing, just like in production. If the agent makes the same mistake again, it will see the same empty result and probably reach the same wrong answer. The case reproduces the failure, not only the input.

The conversion itself is a small script: read the trace, pull out the input and tool results, write fixtures, and leave expect for a person to fill in. I do not let the script guess the expectation. Deciding what "correct" means is the valuable part, and it needs someone who understands the incident.

Grading: check the path, not only the answer

For an agent like this, the final text is only half the story. A run can land on the right cause for the wrong reason, or the wrong cause after doing everything right. I grade at three levels:

A grader for the trajectory part does not need to be clever:

def grade_trajectory(case, run):
    calls = [(c.tool, c.args) for c in run.tool_calls]
    missing = [m for m in case.expect.must_call if not any(matches(m, c) for c in calls)]
    forbidden = [c for c in calls if any(matches(f, c) for f in case.expect.must_not_call)]
    return {
        "pass": not missing and not forbidden,
        "missing": missing,
        "forbidden": forbidden,
    }

When I do need a model judge, for example to decide whether the evidence actually supports the cause, I treat it like any other dependency: pinned model version, a written rubric, and a small set of human-labelled cases to check it still agrees with people.

Running it in CI

Once cases live in the repo, the CI part is familiar:

The evals also produce traces, in the same format as production. When a case fails in CI, you debug it with the same tools you use for a production incident. That symmetry is a large part of why this approach works.

Where the cases come from

After a few months, the eval set looks very different from the hand-written one it started as. The sources I lean on, in order of value:

  1. Negative feedback from users. Every "this was wrong" with a trace attached is a candidate case.
  2. Incidents and near-misses. Anything the agent touched during a real incident gets reviewed, even if it was right. Correct runs during hard incidents make strong positive cases.
  3. Automatic flags. Runs that hit tool errors, timeouts, very long trajectories or low confidence. Cheap to detect, often interesting.
  4. Random samples. A small weekly sample keeps the set honest about ordinary traffic, not only the failures.
  5. Hand-written cases. Still useful for things that have not happened yet, especially safety cases. But they become the minority.

I also delete cases. When a tool is removed or an alert type no longer exists, cases built on it go too. An eval set nobody maintains turns into noise, just like a flaky test suite.

Tradeoffs

This approach is not free.

Checklist

Hand-written evals tell you whether the agent does what you expected. Evals built from traces tell you whether it still gets wrong what it already got wrong once. The second set is the one I would block a release on.