AGENT QUALITY ENGINEERING
AI Agent Evaluation: Metrics, Traces and Release Gates
Evaluate what the agent achieved, how it acted, which tools it used and whether the system stayed inside operating limits.
AI agent evaluation measures both the outcome and the execution path. A useful evaluation system checks whether the task succeeded, whether the agent chose the correct tools and arguments, how state changed, whether the trajectory remained safe and efficient, and whether latency and cost stayed within release limits. Final-answer quality alone cannot expose a faulty process.
What Is AI Agent Evaluation?
AI agent evaluation is the process of measuring whether an agent system works correctly — not just whether the final answer looks right. Because agents take multiple steps, call tools, and modify state, evaluation must inspect both the outcome and the trajectory that produced it.
This separates agent evaluation from traditional LLM evaluation. An LLM produces a single response; an agent produces a sequence of decisions, actions, and state changes. Each step can succeed or fail independently, and the final answer may be correct even when the process was unsafe.
Why Final-Answer Evaluation Is Not Enough
The Agent Trace Inspector above shows three traces. Trace 2 produces a correct answer but through an unsafe trajectory — it called an unauthorized tool and bypassed the approval gate. Final-answer evaluation would pass this trace. Trajectory evaluation blocks it.
An agent that reaches the right answer through the wrong process is not safe to deploy. The process determines what happens when the answer is wrong, when the tool fails, or when the context changes. If you only evaluate the final answer, you are testing the output of a system whose internal behavior you cannot trust.
What Should You Evaluate in an AI Agent?
Six dimensions cover the gap between “the answer looks right” and “the system is safe to release.”
Task Completion and Outcome Quality
Did the agent achieve the user's goal? Measure task success against a defined success criterion — not just “the response sounds good.” For multi-step tasks, define what “complete” means before evaluation.
Agent Trajectory and Step Efficiency
Did the agent take a safe and efficient path to the answer? Trajectory evaluation checks each step: was it necessary, was it correct, did it stay within scope? An agent that takes 15 steps to do what should take 3 is not production-ready, even if the answer is correct.
Tool Selection and Argument Correctness
Did the agent choose the right tool and pass valid arguments? Compare tool calls against expected sequences. Test with invalid arguments, tool failures, and timeouts to verify graceful handling.
State, Memory and Handoff Integrity
Did state remain consistent across steps? In multi-agent systems, did handoffs preserve data correctly? State corruption and handoff failures are subtle bugs that trajectory evaluation can catch.
Groundedness, Safety and Policy Compliance
Is the answer grounded in retrieved data (not hallucinated)? Did the agent violate any safety policies — accessing restricted data, attempting unauthorized actions, or producing harmful content? Policy compliance must be checked per action, not just in the final response. Examine security controls for policy enforcement.
Latency, Token Use and Cost
Did the agent stay within budget? Measure P95 latency, total token consumption, and per-task cost. An agent that is correct but costs 10x the budget is not deployable. Set cost thresholds in the release gate.
End-to-End, Trajectory and Component-Level Evaluation
Three evaluation levels provide different visibility:
- End-to-end: Give the agent a goal, check the final outcome. Fast but blind to process.
- Trajectory: Inspect every step the agent took. Catches unsafe paths, unnecessary steps, and tool misuse.
- Component-level: Test individual components — retrieval quality, tool argument validation, state checkpointing — in isolation. Catches bugs that trajectory evaluation might miss.
Use all three. End-to-end tells you if the system works. Trajectory tells you if it works safely. Component-level tells you why it fails when it does.
Offline vs Online Agent Evaluation
Both are needed. Offline catches regressions before release; online catches real-world issues that offline did not predict.
| Dimension | Offline evaluation | Online evaluation |
|---|---|---|
| When | Before deployment, on curated datasets | After deployment, on live traffic |
| Dataset | Curated, labeled, controlled | Real user inputs, uncontrolled |
| Cost | Compute cost only | Production cost + user impact |
| Catches | Regressions, known failure modes, gate criteria | Distribution shift, new failure modes, real-world edge cases |
| Risk | No user impact | Can affect real users — use canary deployment |
| Action | Block or allow release | Alert, rollback, or patch |
How to Build an Agent Evaluation Dataset
- Define task categories. Group tasks by type: single-tool, multi-tool, multi-step, multi-agent. Each category needs its own success criteria.
- Collect real inputs. Use production logs (sanitized), user-reported issues, and edge cases — not just synthetic examples.
- Label expected outcomes. For each input, define what success looks like: the correct answer, the expected tool sequence, and the acceptable trajectory.
- Include failure cases. Add inputs that should fail gracefully. The agent's handling of failure is as important as its handling of success.
- Version the dataset. Track changes so regressions are measured against a stable baseline. Update the dataset when new failure modes are discovered.
- Keep it representative. The dataset should reflect real usage distribution, not just easy cases. Over-representing edge cases skews results.
Human Review, Code Evaluators and Model-Based Judges
Three evaluator types, each with strengths and limitations:
- Human review: Highest quality for subjective criteria and edge cases. Slow, expensive, and does not scale. Use for high-stakes decisions and dataset creation.
- Code evaluators: Deterministic checks for objective criteria — argument validity, state consistency, tool-call correctness. Fast, repeatable, and scalable. Use for everything that can be checked programmatically.
- LLM judges: Useful for subjective quality dimensions — response relevance, tone, groundedness. Have their own biases and failure modes. Use as a complement, not a replacement, for code and human evaluation.
Combine all three. Code evaluators handle the deterministic checks. LLM judges handle subjective quality. Human review handles the cases where both are uncertain.
Designing a Production Release Gate
A release gate is a set of criteria that must all pass before an agent version is deployed. The thresholds below are illustrative values, not universal standards. Set your own thresholds based on your task, risk tolerance, and production requirements.
| Gate criterion | Illustrative threshold | Block release if |
|---|---|---|
| Minimum task success | ≥ 85% on eval dataset | Below threshold |
| Max invalid tool-call rate | ≤ 2% | Exceeds threshold |
| Safety-policy violations | 0 violations | Any violation detected |
| P95 latency | ≤ 3 seconds | Exceeds threshold |
| Per-task cost | ≤ $0.05 | Exceeds threshold |
| Regression vs baseline | No metric worse by > 5% | Any metric regressed beyond tolerance |
| Human approval for high-impact | 100% of high-impact actions gated | Any high-impact action not gated |
These thresholds are examples. Production thresholds must be set based on your specific task requirements, risk tolerance, and operational constraints.
AI Agent Evaluation Tools and Observability Platforms
Several tools support agent tracing and evaluation. The right choice depends on your framework, deployment, and evaluation requirements — not on universal rankings.
- LangSmith: Tracing and evaluation for LangChain/LangGraph agents. Provides trajectory inspection, datasets, and evaluation pipelines.
- Langfuse: Open-source observability and evaluation platform. Framework-agnostic tracing with evaluation support.
- OpenTelemetry: Standard for distributed tracing. Can instrument agent systems for custom observability pipelines.
- Custom pipelines: For specialized requirements, build evaluation pipelines on top of your existing observability stack. See the architecture control plane for where observability fits.
Common Agent Evaluation Failures
- Final-answer only. Evaluating only the output, not the trajectory. Misses unsafe processes that happen to produce correct answers.
- No failure cases. Testing only success paths. The agent's failure handling is what determines production safety.
- Eval dataset drift. The evaluation dataset does not reflect real usage. The agent passes offline tests but fails in production.
- No cost or latency gates. The agent is correct but too slow or expensive to deploy.
- LLM judge as sole evaluator. Relying on a model to evaluate another model without code-based checks or human review.
- No regression baseline. No comparison against the previous version. Regressions go unnoticed.
Frequently Asked Questions About AI Agent Evaluation
How do you evaluate an AI agent?
What is trajectory evaluation?
How do you test tool calls?
Can an agent give the right answer through the wrong process?
What is the difference between offline and online evaluation?
Should an LLM judge evaluate another AI agent?
What should block an agent release?
Related Agentic AI Guides
RELEASE WITH EVIDENCE
Learn to Release Agents With Evidence, Not Confidence.
Build an evaluation and control platform that scores task success, trajectories, tool use, cost, latency and safety, then applies those results to an engineering release decision.
Reviewed by School of Core AI Technical Training Team
Sources and Methodology
This guide synthesizes evaluation practices from Google Cloud agent documentation, LangSmith, Langfuse, and OpenTelemetry. The release-gate thresholds are illustrative examples, not universal standards. Evaluation methodology must be tailored to your specific task, risk profile, and production requirements.
- Google Cloud — Agent evaluation documentationAgent evaluation patterns and best practices.
- LangSmith — Tracing and evaluationTrajectory inspection, datasets, and evaluation pipelines.
- Langfuse — Open-source LLM observabilityFramework-agnostic tracing with evaluation support.
- OpenTelemetry — Distributed tracingStandard for instrumenting agent systems for observability.
Framework capabilities change. Verify current documentation before implementation. This guide avoids permanent statements such as “Framework X is always best.”