Evaluating an AI agent is a different problem from evaluating a single LLM call. A chat completion succeeds or it doesn't, full stop. An agent runs a multi-step workflow, chaining tool calls across retry state, and a wrong answer in step two doesn't stay contained to step two. It corrupts every step downstream that depended on it being right.
When we built a durable AI code agent at Vercel, that shift changed the question we were asking. We stopped asking whether a given response looked good and started asking whether the workflow succeeded end to end.
We see that same gap across most agent deployments, and the survey data backs it up. 70% of developers are still evaluating outputs by hand, the habit carried over from grading single completions. We evaluate v0 and internal agents differently, through production-like traces, CI gates, checks ordered by cost, and regression tests built directly from failures we've already watched happen in production.
Copy link to headingKey takeaways
The order you reach for evaluation methods matters more than which methods you choose: start with the cheapest check that can catch a given failure.
Binary metrics grounded in observed production failure modes beat generic quality scores; a float invites debate, a pass/fail demands a fix.
Agent evals must grade each intermediate step individually and the full trace as a single unit, because an early error corrupts everything downstream.
An eval suite needs automatic execution to work as an engineering control.
Benchmark contamination can inflate public leaderboard scores, and single-run reporting hides reliability; pass^k, where all k attempts succeed, is the reliability signal those scores hide.
Copy link to headingTreating agents like stateless LLM calls is the core mistake
A standard LLM eval scores one input against one output. Agent evaluation has to score something wider, whether a full workflow succeeds across every intermediate step and every point where the agent has to recover from a bad tool call. That's a structurally different measurement, and scoring an agent like a single completion is the anti-pattern that keeps showing up.
The survey data shows how incomplete that shift still is. Across surveyed teams, 70% of developers evaluate model outputs with manual testing, only 34% use automated evals, and 15% run no formal evaluation at all. The gap comes from a chat-completion habit persisting in agent work: a human eyeballs the final answer, the same check that worked for scoring a single completion, applied to a workflow it wasn't built to grade.
Agent eval tooling already covers this problem. Trajectory evaluation, agentic metrics, red-team workflows, and observation-level scoring all exist and are built for exactly this case. Closing the gap is a matter of adoption, not tooling availability.
In practice, the tempting move is to instrument everything and bolt on an LLM judge for every check. We push back on that instinct. Building v0, the question we ask first is what's the cheapest check that can catch this specific failure. Most of the agent failures we've watched in production, including unresolved imports and missed config changes, are catchable by a deterministic assertion that costs nothing and runs in milliseconds. Broken tool sequences fall into the same category more often than expected. The expensive graders earn their place only where the cheap checks structurally can't see the problem.
Copy link to headingStart with the cheapest check that can catch the failure, then escalate
We start with the cheapest, fastest check that can catch a given failure, and escalate to a more expensive method only when the cheaper one structurally can't see the problem. Get the ordering wrong and the suite gets too slow and too costly to run on every change.
For v0, the first layer is a set of checks with zero token cost and millisecond execution: regex that catches inline styles where Tailwind classes belong, validation that code blocks parse, confirmation that imports resolve, verification of multi-file usage, and the ratio of comments to code as a signal of model laziness. If a check can be expressed as an assertion, it should be. It costs nothing and it never flakes.
Above assertions sit an escalation ladder, each level buying more judgment at the cost of runtime and determinism:
The LLM-as-judge row carries a real cost beyond token spend. Position bias across 21 judges ranged from pb=0.002 for Gemini 2.5 Pro to pb=0.192 for Qwen 3 8B, nearly two orders of magnitude apart, and same-verdict rates that sit above 95% at temperature 0 fall to as low as 70% at temperature 1. That variance is the argument for putting deterministic checks first and reaching for a judge only once the cheaper rows are exhausted.
v0 keeps its metrics binary and grounded in reality: import resolution, rendering, Tailwind class usage, where inline styles would be a failure. These are grounded in failure modes already watched in production. Pass or fail beats a float, because a float invites an argument instead of a fix.
Copy link to headingWhere the ladder breaks down
The ladder fails in one direction more than any other: teams reach for the wrong rung too early. Instrumenting every check with an LLM judge feels rigorous, but it's the same mistake as grading an agent like a single completion, just with more expensive tooling attached to it. An eval suite that costs too much and runs too slowly gets skipped on the changes that need it most, and a suite that stops running on every change has stopped being a control. That's the failure mode that quietly ends most eval programs, not a bad grader but a suite nobody runs anymore.
Copy link to headingAgent evals must grade each step and the full trace as a single unit
Building a durable AI code agent, with code and test generation running inside sandboxed retry loops, single-response grading gave way to step and trace grading. An early error corrupts everything downstream, so agent evals have to grade each step on its own and the full trace as a single unit. A trace where every individual step looks plausible can still end in the wrong state, and a trace with one recovered error can still succeed.
We ran this in practice testing AGENTS.md against skills. Two prompt strategies went head-to-head: invoke the skill first, then explore the project, or explore first, then invoke. In the 'use cache' directive eval, the invoke-first approach wrote the correct page.tsx, but completely missed the required next.config.ts changes. Explore-first got both. Small wording changes that produce large behavioral swings are a sign that the approach is too brittle for production.
Our product design skill evals formalize this into a rubric methodology: an agent edits a before state, a judge checks results against a rubric, holdout examples hide their expected edits to test whether guidance generalizes, and fixtures run without the skill to measure whether it changed behavior at all. Trajectory grading also has to allow valid alternative paths. Exact-sequence matching punishes an agent for taking a different correct route, so presence-based matching asks whether the necessary tool calls happened and whether incorrect calls stayed out of the trace.
Single-run headline scores hide reliability across runs. Function-calling agents can clear well under half of a benchmark's tasks in a single attempt, and that success rate drops further once the bar becomes passing every attempt in a multi-try trial rather than just one. Fixed public test sets need contamination controls too, because models can overfit to static benchmarks and public tasks can leak into pre-training corpora. A single pass leaves reliability unproven.
Copy link to headingApplying the ladder to the AGENTS.md eval
The AGENTS.md eval is a clean case for why step grading matters more than final-answer grading. Final-answer grading alone would have passed the invoke-first run, since the produced page.tsx was correct. A presence-based assertion, the cheapest rung on the ladder, checking whether next.config.ts was touched at all, would have caught the miss instantly and for free. No judge needed, because the failure was structural, not a matter of interpretation. That's the pattern worth carrying forward: before reaching for a judge to score trajectory quality, check whether a presence-based assertion already answers the question.
Copy link to headingEvals need CI to work as engineering controls
An eval suite needs automated enforcement to work as an engineering control. The pattern we recommend and use is a loop: score sampled production traffic automatically, send the remainder to human review, and feed results back into prompts and guardrails.
CI integrations handle enforcement directly. Promptfoo's GitHub Actions workflow reads results.stats.failures and exits with code 1 when failures exceed zero, which blocks the merge. LangSmith's pytest and Vitest integrations let evals run inside the testing frameworks teams already use.
Preview deployments extend the same gate to full-application behavior. Stably's agents run autonomous end-to-end tests against Vercel preview URLs, read code diffs, and validate whether changes actually work, cutting agent shipping time from weeks to hours.
Manual testing still dominates the field: that same 70/34 split from earlier, most teams checking outputs by hand and a minority running automated evals. An eval suite treated as a one-time offline activity stops being useful the moment a prompt or model changes, especially once the underlying data sources shift too.
Copy link to headingEach eval tool covers a different slice of the problem
Tool choice is under active reconsideration right now. OpenAI deprecated its Evals platform on June 3, 2026, with a full shutdown on November 30, 2026, and the official migration path points to Promptfoo. Teams re-evaluating should map tools to the slice of the problem each one actually solves.
Braintrust stands apart among these tools because it's the only framework with a native Vercel Marketplace integration and automatic Vercel AI SDK tracing. It was also our collaborator on the Bash agent eval, where the Bash agent underperformed both the SQL agent and even basic filesystem tools on task accuracy, and shared trace visibility between the two teams improved both the tools and the benchmark. The framework matters less than the traces it gives access to. Every tool in this table can run a score. The differentiator is how quickly a production failure becomes a regression test.
Copy link to headingEvals compound when grounded checks are always running
On error-free generation evals, v0's composite architecture substantially outperforms the general-purpose frontier models underneath it on the same web-development tasks. That gap is what eval-driven development produces over time. The AutoFix model in the v0 pipeline trained on reinforcement fine-tuning and a large volume of real generations, and its evals and user feedback tracked patterns and flagged where outputs needed correction. Every failure the evals caught made the next generation cheaper to fix.
Those principles hold across frameworks: order checks from cheapest to most expensive, escalate only when the cheap check structurally can't see the failure, ground binary metrics in observed production failures, and enforce evals in CI so they run on every change. Production sampling should feed new failures back into the suite the same way.
The gap is still wide, with most teams evaluating by hand rather than automating. What separates a team stuck at manual review from one running the v0 AutoFix loop isn't a smarter grader. It's whether every production failure gets converted into a cheaper failure to catch next time.
Copy link to headingWhat's the difference between grading a step and grading a trace?
Step grading checks whether each individual action in a workflow, a tool call, a file edit, a retry, was correct on its own. Trace grading checks whether the full sequence reached the right end state. Both matter because a trace can look correct step by step and still land in the wrong state, and a trace with one recovered error in the middle can still succeed overall.
Copy link to headingWhen should you use an LLM judge instead of a deterministic assertion?
Only once a deterministic assertion structurally can't see the failure. Assertions catch things like unresolved imports, missed config changes, and broken tool sequences at zero token cost. An LLM judge earns its place for judgment calls an assertion can't make, trajectory quality or tone, and it comes with real cost: position bias that varies by two orders of magnitude across judges, and same-verdict rates that drop from over 95% to as low as 70% depending on temperature.
Copy link to headingWhat is pass^k, and why doesn't a single successful run prove reliability?
Pass^k measures whether all k attempts at a task succeed, not just one. A single successful run can hide a low underlying reliability rate. Agents that clear roughly half a benchmark's tasks on a single attempt can see that rate fall sharply once every attempt in a multi-try trial has to succeed. Reporting only a best-case single run obscures that gap.
Copy link to headingHow do you keep an eval suite fast enough to run on every change?
Order the checks by cost. Deterministic assertions run in milliseconds and cost nothing, so they run on every change without slowing anything down. Expensive graders like LLM judges get reserved for the failures that cheap checks structurally can't catch. A suite that puts expensive checks first gets too slow to run consistently, and a suite that doesn't run consistently stops functioning as a control.
Copy link to headingHow do Vercel's eval tool integrations export AI SDK traces?
Vercel AI SDK traces integrate with eval and observability tooling through native exporters. Braintrust gets automatic AI SDK tracing through its Marketplace integration. LangSmith and Langfuse both use an AI SDK exporter. Arize Phoenix and other OpenTelemetry-compatible backends can receive AI SDK traces exported through OpenTelemetry directly.