AI Agent Evaluation

What Is AI Agent Evaluation?

AI agent evaluation is the practice of measuring whether an AI agent completes tasks correctly end to end, scoring the full trajectory of decisions, tool calls, and intermediate steps as well as the final outcome. It extends single-response evaluation to multi-step autonomous work, where an agent can reach a right answer through a broken process or fail ten steps after a good start. The stakes keep rising because agents keep getting more capable: METR measured in 2025 that frontier models like Claude 3.7 Sonnet have a 50 percent task-completion time horizon of about 50 minutes, a horizon that has been doubling roughly every seven months since 2019 [1].

Key Takeaways

• Agents need trajectory-level evaluation, not just output-level. A correct final answer reached by deleting and recreating a customer record is still a failure.
• The three standard layers are outcome (did the task complete), trajectory (were the steps sensible and safe), and tool use (right tool, right arguments, results actually used).
• Small per-step error rates compound: 95 percent step accuracy across twenty steps yields roughly a one-in-three task failure rate, which is why agents feel less reliable than the models inside them.
• Environment design is half the work. Reproducible sandboxed environments with checkable end states make agent evals trustworthy; ad hoc demos do not.
• Pair offline evals with production tracing. The eval suite predicts behavior; AI observability confirms it against real traffic.

How It Works

Evaluating an agent means evaluating a loop, not a response. The agent receives a task, plans, calls tools, reads results, and iterates, so the evaluation has to capture the whole trace. Outcome checks are the anchor: define a verifiable end state, the ticket is closed with the right resolution code, the pull request passes tests, the row exists in the database, and assert it programmatically. This is the agent equivalent of an acceptance test, and it is what most published agent benchmarks like SWE-bench-style suites measure. Those benchmarks also show how fast the ground moves: when SWE-bench launched in 2023, the best-performing model, Claude 2, resolved only 1.96 percent of real GitHub issues end to end [2], and the Stanford HAI 2025 AI Index recorded SWE-bench scores rising 67.3 percentage points in a single year [3].

Outcome checks alone hide dangerous behavior, so mature suites add trajectory grading. Here you score how the agent got there: number of steps versus a reasonable baseline, whether it called destructive tools it did not need, whether it recovered from a failed tool call or looped on it, whether it invented tool arguments. Some of this is rule-based (assert the delete endpoint was never called), and the qualitative rest goes to an LLM judge reading the trace against a rubric. Tool-use metrics sit underneath: correct tool selection rate, argument validity, and whether returned data actually influenced the next step.

The infrastructure requirement that surprises teams is the environment. Each eval case needs a reproducible starting state, a seeded database, a fixture repository, a mocked API, so runs are comparable across model and prompt changes. Because agents are nondeterministic, single runs mislead; running each case several times and reporting pass rates (the pass@k pattern borrowed from code generation research) gives numbers you can trust.

Example

A team builds a support agent that handles refund requests end to end: look up the order, check the refund policy, issue the refund through the payments API, and email the customer. Their eval suite contains 60 scenarios, each a seeded sandbox with a fake order database and a mocked payments API. One scenario: an order outside the 30-day window. The outcome check asserts no refund was issued and the customer email explains the policy. The trajectory check asserts the agent consulted the policy tool before deciding, rather than guessing. When they trial a cheaper model for the inner loop, outcome pass rate holds at 92 percent, but trajectory grading shows it skipping the policy lookup and answering from memory in a quarter of runs, right answers by luck. They keep the cheaper model but add a guardrail forcing a policy lookup before any refund decision, and the shortcut disappears.

What People Get Wrong

The classic mistake is evaluating the agent's chat transcript instead of its effects. Teams read a polished conversation, see the agent claim it updated the CRM, and score it a pass without checking the CRM. Agents confidently narrate work they did not do, or did wrong, and transcript-level review misses exactly those cases. Ground truth lives in the environment: assert on the database row, the API call log, the file diff. If your eval cannot check the end state programmatically, it is a vibe check with extra steps.

FAQ

How is agent evaluation different from LLM evals? Standard LLM evals grade one input against one output. Agent evaluation grades a sequence: planning, tool calls, state changes, and recovery behavior across many model invocations, judged against an environment's end state. The single-response methods still apply inside each step, but they are necessary rather than sufficient.

What metrics should an agent eval report? Task success rate is the headline, ideally as pass@k across repeated runs. Below it: step efficiency against a baseline trajectory, tool-call error rate, unsafe-action count (which should gate release at zero), cost per task in tokens, and latency. Cost matters more than teams expect, since an agent that succeeds by brute-forcing forty steps may be unshippable economically.

Do multi-agent systems change the approach? The layers stay the same but attribution gets harder. When an orchestrator delegates to specialist agents, you evaluate the system end to end first, then per-agent handoffs to find where failures originate. Trace structure from your orchestration layer becomes the raw material for both.

Sources

1. METR. "Frontier models show a 50% task-completion time horizon of about 50 minutes, doubling roughly every seven months since 2019." https://arxiv.org/abs/2503.14499. Accessed August 2026.
2. Jimenez et al., Princeton. "SWE-bench launch results: best model (Claude 2) resolved 1.96% of real GitHub issues." https://arxiv.org/abs/2310.06770. Accessed August 2026.
3. Stanford HAI. "2025 AI Index Report: SWE-bench scores rose 67.3 percentage points in one year." https://hai.stanford.edu/ai-index/2025-ai-index-report. Accessed August 2026.