Evaluating an AI agent before production means combining two things: deterministic tests that verify specific behaviors, and evals that judge non-deterministic output quality and the trajectory — the sequence of tool calls and decisions — that produced it. Graders can be code-based, model-based (LLM-as-judge), or human, and should be run by someone other than the agent’s own author. Because a single successful run doesn’t prove reliability, teams typically run multiple trials and track metrics like pass@k (the odds of at least one success) rather than trusting one good demo. Just as important, the depth of evaluation required isn’t uniform: an agent’s pre-production bar should scale with the cost of it being wrong — a documentation agent and one that touches payments or personal data shouldn’t clear the same gate before shipping.
A demo that runs clean in front of stakeholders tells you the agent can work but can’t promise it will keep working once real users start pushing edge cases at it. Anthropic and Databricks have already covered the core vocabulary — task, trial, grader — in detail, and engineering leads researching how to evaluate AI agents before production have usually seen it. What those resources don’t get into is the narrower question of LLM agent evaluation at the actual point of deciding to ship.
Take a system that pulls customer order data and refunds people automatically. The real question isn’t how much evaluation an agent needs in the abstract — it’s whether this specific agent, doing this specific task, is safe to turn loose on real accounts. That’s what this article actually answers: not a general number, but how to work out the right bar for the agent in front of you.
Tests vs. Evals: The Two Different Questions a Pre-Production Review Has to Answer
A pre-production review has to answer two separate questions, and conflating them is where most “it works” claims fall apart. The first is deterministic: given a task — a specific input paired with an expected result — does the agent produce the right output. Tests answer that question. They’re checked by code, which makes them cheap to run — but that same rigidity is the catch. Feed a test input phrased slightly differently from what the author intended, and it can fail something that actually worked fine. Tests also handle single-turn and multi-turn agents differently. A single-turn agent produces one output, so there’s one clear pass-or-fail check. A multi-turn agent is harder: a mistake made on turn two might not cause a visible problem until turn seven, so the test only catches it once the damage is already done — not at the point where it actually happened. That’s the gap evaluation methods for multi-turn AI agents have to close: catching the failure at the turn that caused it, not just the turn where it surfaced.
The second question is harder to pin down. Take an agent that summarizes a support ticket about a customer whose payment failed twice this month. The summary it writes is accurate — every fact checks out — but it buries the failed-payment detail in the third paragraph instead of the first line, where a human agent would have put it. Nothing about that shows up as wrong in a test. A grader has to actually read it and decide whether burying that detail matters, which is a judgment call, not a lookup.
That’s the split behind tests vs. evals, and it’s also why traditional software testing doesn’t transfer here: feed the same input twice into old software and you get the same result back to check against. Feed an agent the same ticket twice and there’s no fixed answer waiting — an evaluation framework has to score whether the agent did right by the customer, not whether it matched something written down in advance.
Grader Type | Strengths | Weaknesses |
|---|---|---|
Code-based | Fast, cheap, objective | Rigid — misses subjective quality |
Model-based (LLM-as-judge) | Flexible, scalable, captures nuance | Non-deterministic, needs calibration |
Human | Gold-standard judgment | Expensive, slow, doesn’t scale |
Grader Type
Code-based
Model-based (LLM-as-judge)
Human
Strengths
Fast, cheap, objective
Flexible, scalable, captures nuance
Gold-standard judgment
Weaknesses
Rigid — misses subjective quality
Non-deterministic, needs calibration
Expensive, slow, doesn’t scale
Someone ran the test suite on the refund agent and every test came back green. Nobody in that meeting could tell you whether the agent actually picked a sensible way to handle the edge case where a customer wants a partial refund split across two payment methods — the tests never checked for that, because nobody wrote one for it.
Output Evaluation vs. Trajectory Evaluation: What to Actually Check
Did the agent answer the prompt correctly? That’s the only question output evaluation asks. It’s the same logic as end-to-end testing: feed in the same input, expect the same result back, check if you got it. AI agents break that check. Run the same prompt twice and the path the agent takes can differ even when the final answer doesn’t — output evaluation has no way to see that, because it only ever looks at where the agent landed, never how it got there.
Say the refund agent needs the customer’s current balance. It calls the balance API, gets back a stale number because of a caching bug that has nothing to do with the agent itself, and calculates a refund off that stale number — which happens to match what a fresh balance would have said anyway, pure coincidence. The agent’s final response goes out correct. Nobody reading it would guess a wrong tool call fed into that number a step earlier. That’s specifically what reading the execution trace is for: the actual call it made, the actual number it got back, because steps in a multi-step agent can fail independently of whatever line ends up on screen at the end.
An agent needs a customer’s loyalty tier, so it calls getLoyaltyStatus(). That function hasn’t existed since a refactor eighteen months ago. It gets an error, tries again, gets the same error, tries once more, gives up, and quietly defaults the customer to standard tier — nothing in the final response mentions it couldn’t check. The stale-balance bug from earlier is the same kind of thing, and so is an agent that decides on turn one a customer’s issue is a shipping delay when it’s actually a billing dispute, then spends the rest of the conversation chasing tracking numbers for a problem that was never about shipping. Databricks has documented this class of failure in more depth.
Three lines in a caching function. That’s all it took. Nobody had touched that function in months, and when someone finally did, on a random Tuesday afternoon, the fix didn’t even touch the part that actually broke — by Thursday the agent was quoting order status off data from two messages back, the cart from before the customer changed their mind about size, and QA never caught it because QA was testing the thing that got fixed, not the thing that got broken by fixing it. What actually broke lived inside the agent harness — execution, memory, tools, all of it. The evaluation harness, the part that’s supposed to catch exactly this, sat completely separate: trials, transcripts, grading, running the whole time and passing everything, because the transcripts it read looked fine. Our guide on why AI-generated code breaks in production gets into this specific failure mode in more detail.
The evaluation harness is supposed to catch this. It stays separate on purpose — running trials, grading transcripts on its own timeline, watching but never touching the system itself. That separation is what makes tests different from evals: a test checks whether an answer matches a known-correct one, while an eval has to ask something harder, since often there’s no fixed answer sitting there to check against. What it needs instead is a working definition of “done” — did the task actually get completed — because that’s the only kind of yardstick that survives contact with a system that doesn’t have one right answer.
Good evaluation methods should measure the agent’s ability to produce correct behavior under human oversight, not just whether one grader marked a response as acceptable.
Capability Evals vs. Regression Evals: Which Bar Actually Applies Before Shipping
The split-payment eval — can the agent handle a customer paying with three different cards on one order — failed on the first nine attempts anyone ran it. Someone flagged that as broken. It wasn’t; it was a capability eval doing exactly what a capability eval is supposed to do this early, since splitting a charge across three separate payment methods is a genuinely hard thing to get right, and nobody expected it to work on attempt ten either. What matters is that it’s not the number gating a release — the release gate is the regression suite, sitting somewhere else entirely, checking whether the agent still handles the one-card case it’s always handled.
Regression evals measure whether the agent consistently completes core tasks that previously worked without error. Robust evaluation strategies keep the same quality bar across agent versions, with a clear primary metric for task completion, targeted checks of agent performance, and regression testing before deployment; unlike online evaluations that inspect production traces through production monitoring after launch, these pre-release checks focus on catching failures before users see them. Regression suites demand near 100% pass rates before deployment, ensuring that recent model updates, prompt adjustments, or tool additions have not quietly broken established behaviors. A drop in regression scores indicates system degradation, signaling an immediate block on deployment.
That checkout agent’s split-payment score climbing from nine failures to eight didn’t tell anyone whether it was safe to ship. What told them that was the regression suite — the one checking whether the agent still handled a single-card purchase correctly, the case it had nailed for months. If that number ever dipped, even by one point, nobody cared how the split-payment eval was trending; the release got held. Capability scores earn their way into that regression suite eventually — once split-payment handling stops being the hard new thing and just becomes something the agent isn’t allowed to forget how to do — but that’s a later conversation, not the one happening at release time. Our analysis of verified velocity looks at this same tension from the delivery-speed side, where teams chasing throughput numbers can bury exactly this kind of regression under a faster release cadence.
Matching Eval Rigor to Risk: Not Every Agent Needs the Same Bar
A team ships an eval checklist for their new expense-report agent, then six weeks later reuses that exact checklist, unedited, for a second agent that approves outgoing wire transfers over $10,000. Nobody made that decision on purpose — it’s just the checklist that already existed. The expense-report agent occasionally mislabels a category and someone fixes it in two minutes. The wire-transfer agent has never actually made an error the checklist would catch, because the checklist was never built to check for the thing that would actually be expensive: an amount going to the wrong account.
The wire-transfer agent and the expense-report bot came out of the same three-week sprint, same two engineers, same afternoon of code review. Nobody treated one as more important to get right than the other while building them. What changed is what happens after: someone reads every single output the wire-transfer agent produces before it goes anywhere near a bank, and nobody reads the expense-report agent’s output unless a number looks obviously wrong. A miscategorized lunch receipt gets caught eventually, shrugged off, fixed in the next pass — nobody’s job depends on it. Money leaving the wrong account doesn’t get a next pass.
Zone | Typical Task Type | Minimum Pre-Production Bar |
|---|---|---|
Zone 1 — Always human-led | Architecture, security, payments, personal data, public API contracts | Saturated regression suite plus mandatory independent human sign-off before every release — no exceptions regardless of eval scores |
Zone 2 — AI-assisted, human-verified | Features, integrations, workflows with real but recoverable stakes | Regression suite near 100% plus a calibrated LLM-as-judge for output and trajectory quality, reviewed by an accountable engineer on each release |
Zone 3 — Automated, machine-verified | Documentation, routine maintenance, low-stakes internal tooling | A strong automated regression suite is sufficient — earns full autonomy specifically by being cheap to verify and easy to undo |
Zone
Zone 1 — Always human-led
Zone 2 — AI-assisted, human-verified
Zone 3 — Automated, machine-verified
Typical Task Type
Architecture, security, payments, personal data, public API contracts
Features, integrations, workflows with real but recoverable stakes
Documentation, routine maintenance, low-stakes internal tooling
Minimum Pre-Production Bar
Saturated regression suite plus mandatory independent human sign-off before every release — no exceptions regardless of eval scores
Regression suite near 100% plus a calibrated LLM-as-judge for output and trajectory quality, reviewed by an accountable engineer on each release
A strong automated regression suite is sufficient — earns full autonomy specifically by being cheap to verify and easy to undo
A Zone 1 deployment touching payment processing or user data requires saturated regression testing alongside mandatory human sign-off prior to release. Tracking the same primary metric across agent versions is what makes performance comparable release over release, especially on tasks with a clear ground truth to check against — that’s where evaluation actually reaches the critical components driving reliability, deep enough to catch what a surface-level output check would miss. Conversely, a Zone 3 agent generating internal documentation can ship cleanly on automated regression passes alone, as low recovery costs make intensive manual reviews unnecessary.
None of this replaces watching agent behavior once the system is live. One gates a release, the other watches what happens after — that’s the key insight separating pre-production evaluation from ongoing monitoring. Mapping these boundaries aligns with our recommended path from MVP to production in 90 days, ensuring safety measures scale naturally with system complexity.
Who Signs Off in LLM Agent Evaluation: Why the Agent’s Author Shouldn’t Grade Its Own Work
Good eval engineering starts with independence: an evaluation pipeline remains inherently vulnerable to confirmation bias when managed entirely by the engineers who wrote the prompt logic. Developers who build an agent naturally construct test cases around their intended implementation paths, unintentionally creating blind spots around edge cases and failure modes. Maintaining objective evaluation requires separating system development from final sign-off authority.
A support agent’s rubric rewards short, upbeat replies. For four months, it earns exactly what it should. Then the company adds a mandatory disclaimer paragraph to every return-related reply, and the rubric is never updated to require it. Nothing about the judge changes. It scores replies the same way as before — tone, speed, resolution — whether or not the disclaimer shows up. A reviewer finally reads through a batch of transcripts and finds the disclaimer missing in every one, each still marked as a pass.
Calibrated autonomy isn’t just a technical-complexity question. How much human oversight a system needs tracks the cost of getting it wrong — the more expensive a failure, the more a human has to be in the loop when one happens.
Serhii Leleko
AI & ML Engineer at SPD Technology
“We’ve seen an LLM-as-judge score hold steady for months while the thing it was supposed to be measuring quietly drifted — the grader kept agreeing with itself long after it stopped agreeing with what a human reviewer would actually accept. Calibration against human judgment isn’t a one-time setup step; it’s the only way you find out your judge has drifted before a bad release does.”
Establishing independent sign-off mirrors the core principles of separation of duties found in enterprise software governance, and the broader evaluation strategies for calibrated autonomy should also include Safety, Trust, and Guardrails checks for compliance with business logic and ethical rules.
Our analysis of spec-first development for AI-assisted engineering demonstrates how external validation prevents circular assumptions from reaching live environments.
Applying independent sign-off ties release approval to objective performance metrics that a second reviewer can verify on their own, separate from how confident the original author feels about it.
Non-Determinism and AI Agent Evaluation Metrics: Why One Good Run Isn’t Enough
Agent behavior varies between runs on the same task, even when nothing in the prompt or the code changes. A single successful demo run and a genuinely unreliable agent can look identical until someone runs the task more than once, which is exactly why non-determinism sits at the center of any serious pre-production check. For a conversational agent, that variability is even harder to judge because multi turn interactions can hide failures until a later step in the same agent run.
Anthropic defines two of the most important AI agent evaluation metrics for this purpose: pass@k and pass^k.
- Pass@k is the probability an agent succeeds at least once across k trials, and it rises as k grows — the right metric when one success is enough, which covers most coding tasks at pass@1.
- Pass^k is the probability that all k trials succeed, and it falls as k grows — the metric that matters when consistency itself is the requirement, such as a customer-facing agent that can’t afford an inconsistent answer on a later attempt.
In practice, agent performance should be reviewed against a primary metric tied to the business goal, whether that is task completion for a customer support agent or answer accuracy in a workflow where every successful task must also stay within acceptable token usage and cost.
The math makes the stakes concrete: an agent with a 75% per-trial success rate has roughly a 42% chance of succeeding on all three of three trials (0.75³). That number is worth sitting with before treating “it worked when I tried it” as evidence of much of anything.
In other words, LLM-as-a-judge can help score repeated trials at scale, but calibration still matters because how an agent performs should be checked against human review on a recurring basis. That calibration should also consider user experience metrics such as user satisfaction, task abandonment, and time to resolution when those are the real business outcomes.
The Pre-Production Eval Gate: A Practical Eval Engineering Checklist
Before releasing agent workflows to live traffic, engineering teams should evaluate their test coverage and governance frameworks against a structured readiness gate. This is where evaluating AI agents before production stops being theoretical and becomes a checklist a team can actually run against a specific release. Multi-turn interactions are harder to debug because context loss across turns can hide why the agent behaves inconsistently, so eval data should preserve relevant context for reproducible review.
-
Deterministic tests and non-deterministic evals both exist for this agent, with effective evaluation based on explicit evaluation strategies for offline checks and online evaluations after launch
-
Both output and trajectory are evaluated, not output alone, including tool usage and whether production traces let teams monitor agent behavior safely without exposing sensitive data
-
A regression suite exists and sits near 100% pass rate, with production monitoring in place to catch drift and failures in live use
-
The agent’s task has been mapped to a calibrated-autonomy zone
-
Sign-off comes from someone other than the agent’s author
-
LLM-as-judge graders, if used, are calibrated against human review
-
The agent has been run across multiple trials, not judged on one pass; while answer accuracy may be the primary metric for simple tasks, customer support workflows are often judged by task completion or user satisfaction across an agent run
-
A team’s written down what counts as done here — not just “did it get the right answer,” but whether the person on the other end actually got helped and didn’t give up halfway through.
Failing to check the sign-off or trajectory rows usually signals a missing governance process — an accountability gap nobody has assigned yet. Fixing organizational alignment often proves faster and cheaper than adding extra testing infrastructure. A team should also know roughly what a run costs before it ships — not a full accounting, just enough to know if the agent is burning ten times the tokens it needs to for the job it’s doing.
Reviewing our AI Production-Ready Checklist helps teams catch the things this list doesn’t cover — access controls, rollback plans, who gets paged at 3am if something breaks.
Our Expertise
SPD Technology built an AI incident-management system for a US fintech/SaaS platform that triages production issues and drafts PR-ready fixes on its own. It resolves roughly 70% of incidents without a person touching them, and for the fixes it does draft, response time dropped from over 60 minutes to under 30 — enough that the client no longer needs round-the-clock on-call engineering coverage. The other 30% aren’t a shortfall in the system. They’re cases the eval gate flagged its confidence threshold, so a human gets pulled in before anything ships — which is the point of building it this way.
Our engineering teams build and deploy resilient agentic architectures by applying strict operational standards:
- Careful eval engineering keeps capability and regression tests in separate suites, so teams immediately distinguish new functional breakthroughs from unexpected regressions.
- Both output and trajectory are evaluated, not output alone.
- Monitoring tools log every action in each agent run for effective debugging.
- Sign-off authority for AI-generated code changes remains strictly with an accountable engineer who did not author the original prompts or codebase.
- SPD Technology treats LLM agent evaluation as risk-scaled by default: pre-production evaluation thresholds scale directly with task risk zones, preventing low-impact tools from being over-engineered while enforcing strict safety checks on high-stakes systems.
- Model-based evaluation tools undergo routine human calibration checks, preventing judge drift from degrading the accuracy of automated quality metrics — the same drift that, left unchecked, eventually shows up in platform health metrics every scaling CTO should track, once a system is live.
- There’s a plan for watching this thing after it ships — not just whether it’s still running, but whether it’s still doing the right thing, with customer data handled carefully whenever someone reviews a real trace.
Conclusion
A prescription-refill agent that approves nine refills correctly and misses a dosage conflict on the tenth looks fine right up until it doesn’t. That gap — between what a demo shows and what happens once real patients are on the other end of it — is what how to evaluate AI agents before production is really about. Last week’s fixes have to still be holding, not just this week’s new feature working. Someone has to actually sit down and read what the agent did, step by step, instead of glancing at whether the final answer looked right. And whoever signs off can’t be the person who wrote the prompt. None of that shows up in a five-minute demo, and none of it happens by accident — it’s a checklist sized to what a specific agent does and what a mistake with it actually costs, run before that agent goes anywhere near a real patient.
Key Takeaways
- Deterministic unit tests verify code execution, while non-deterministic evals assess answer quality and system decision paths.
- Evaluating final outputs without inspecting execution trajectories risks hiding hallucinated tool calls, stale memory errors, and inefficient reasoning loops.
- A capability score can climb every release while something that used to work quietly stops working, and nobody notices unless the regression suite is actually being watched too.
- How much scrutiny an agent needs before shipping isn’t fixed — it tracks what a mistake actually costs, which is why a payments agent needs a human checking every release and a low-stakes one doesn’t.
- Allowing system authors to grade their own agent evals introduces confirmation bias, hiding judge calibration errors and grading bugs.
- Statistical variance degrades multi-run consistency, turning a 75% single-run success rate into a 42% probability of three consecutive successful executions.
- Model-based evaluation judges experience performance drift over time, requiring routine calibration against human expert benchmarks.
- In short: an agent earns production readiness by clearing a bar sized to what it does, not by surviving a demo.
FAQ
How many eval tasks do I need before shipping an AI agent?
Say a team is scoping an eval suite for an early invoicing agent. They land on 34 tasks, most pulled straight from support tickets where the old manual process had gone wrong before. That’s roughly the right range at this stage — a couple dozen to fifty, not more — because almost any change to the agent shows up clearly in a run this small; padding past that doesn’t buy much yet. One of their early tasks just asked whether an invoice summary was “reasonably accurate.” Two reviewers scored the same output differently on three separate runs, because “reasonably accurate” meant something different to each of them. It only stopped producing noise once someone rewrote the task with an actual expected total attached — a number either matched or it didn’t. Once the agent’s been live long enough that a single bad release stops being obvious on sight, the suite has to grow — past a hundred tasks, usually — because that’s what it takes to keep catching regressions small enough to hide.
What’s the difference between output evaluation and trajectory evaluation?
Did the agent land on the right answer — that’s what output evaluation checks. It says nothing about how it got there. Trajectory evaluation looks at that path instead: the full record of tool calls, reasoning, and intermediate steps, because an agent can stumble into a correct result through a route that’s inefficient, risky, or just lucky. Hallucinated tool calls, stale memory, an agent committing early to a wrong assumption and never recovering — these are the trajectory-level failures a technically correct final output can hide.
Should the same team that builds an AI agent also grade its evals?
They can write the evals, but grading the results and deciding whether they’re sufficient to ship should involve someone other than the agent’s author. An agent’s own team reading its own passing scores as sufficient is a closed loop that misses grading bugs, miscalibrated LLM-as-judge criteria, and cases where an eval technically passed for the wrong reason. Independent review — a second engineer, a dedicated evals owner, or a structured sign-off process — is what catches these before they reach production.
What is LLM-as-judge and can it be trusted for pre-production decisions?
LLM-as-judge means using a model to grade another model’s or agent’s output against a rubric, which scales well and captures nuance that rigid code-based checks miss. It can be trusted for pre-production decisions, but only once it’s been calibrated against human review — meaning a human has confirmed the judge’s scores align with what a person would actually conclude on a representative sample of cases. An uncalibrated LLM-as-judge can produce confident, consistent-looking scores that have quietly drifted from what actually matters, which is why teams need to keep recalibrating on a set schedule — a single setup pass won’t catch drift that shows up months later.
How do capability evals and regression evals differ, and which one gates a release?
Capability evals measure what an agent can newly do and are supposed to start at a low pass rate — a target to improve against as the system matures. Regression evals measure whether an agent still reliably does everything it used to, and should sit near 100%; a drop signals something broke. In practice, a regression suite near saturation is the metric that should actually gate a release — a strong capability score with no regression coverage tells you nothing about whether the change broke existing behavior.
Does a low-risk AI agent need the same evaluation rigor as a high-risk one?
No, and treating them identically usually means either wasting effort on a low-stakes agent or under-testing a high-stakes one. A conversational agent handling multi turn exchanges often needs more review than a single-turn one, since losing context across turns makes debugging harder to do reliably. The depth of pre-production evaluation should scale with what a wrong action would actually cost: an agent operating in a low-stakes, easily-reversible area can reasonably ship on a strong automated regression suite alone, while one touching irreversible decisions, payments, or personal data needs both a saturated regression suite and mandatory independent human sign-off, regardless of how good its eval scores look.
How much does a pre-production AI agent evaluation review cost, and how long does it take?
Determining the cost and timeline for an evaluation review depends on system complexity, the number of risk zones involved, and the maturity of existing testing pipelines. A low-risk Zone 3 agent needs only a narrow automated review. A Zone 1 system touching financial or personal data is a different scope entirely — one where the review has to validate the sign-off process as much as the eval coverage itself.