A benchmark can tell a team which agent solved more tasks last month. It cannot tell them whether their reviewers can absorb what that agent produces, whether it follows their conventions, or whether passing tests means anything. Those are the questions that decide the outcome. This guide covers five criteria that answer them, tested on real tickets in a real repository.
Copy link to headingKey takeaways
The unit of evaluation is the harness rather than the model, so the testable questions are whether a team can choose the model, assign different models to different steps, and bring its own key.
Pick the highest autonomy level the available review capacity absorbs, because the emerging mispractice is level-skipping: operating nominally with humans responsible for review while merging agent changes nobody meaningfully inspected.
Context selection quality decides outcomes more than window size, since unfiltered or wrongly selected context delivers limited or negative benefit.
Repository awareness shows up fastest in the reuse test, where a weak agent reimplements an existing helper under a different name instead of calling it.
An agent that writes both the implementation and the tests produces passing tests that show consistency rather than correctness, so self-verification needs an independent check.
Agent pull requests cost reviewer attention rather than quality. In the 2× mandate study, they took about 20% longer from first human review to merge, while merge and revert rates held steady.
Copy link to heading5 criteria for evaluating AI coding agents at a glance
The five criteria are autonomy level, context handling, repository awareness, self-verification, and review and merge workflow, and they all draw on the same limited budget. An agent with broad autonomy and thin repository awareness opens more pull requests than a team can review with confidence, which is how reviewer attention becomes the currency the first four criteria spend.
Each criterion has a concrete test and a failure signal a reviewer can name:
Read the table as a sequence rather than a checklist. Test autonomy first, because the level a team picks determines how much of everything else matters, then context and repository awareness on the same tickets, since both surface in the diffs the first test already produced. Self-verification and the review queue need production conditions, so they run last and take the longest to read.
Score each criterion against the team's own baselines rather than an absolute bar. Time to PR, acceptance rate, and rework rate for human-authored changes in the same repository are the comparison that matters, and a two-week trial with three to five engineers on real tickets produces enough of them to compare. Architecture and conventions drive the result more than public benchmark tasks do, which is why a leaderboard position transfers poorly.
When the criteria disagree, weight the one closest to merge. A slower agent whose pull requests clear review at the existing baseline rate delivers more than a faster agent whose pull requests stall, and an agent that scores well everywhere except repository awareness is the most expensive of the failure modes, because duplication surfaces after the trial ends.
Copy link to headingCriterion #1: Set the coding agent's autonomy level
Autonomy level is how much of the plan, edit, test, and ship loop the agent runs unsupervised. A position paper from Berkeley's Center for Responsible, Decentralized Intelligence proposes three levels of software-development autonomy, modeled on the Society of Automotive Engineers (SAE) driving-automation levels, and defined by which stages have moved from human responsibility to AI control.
The levels separate systems that get described with the same words:
Level I, code autonomy: The agent owns design and implementation without line-by-line approval and returns a complete pull request with rationale, code, and documentation. Humans still decide what to build, review at pull request granularity, oversee testing and security auditing, and gate deployment.
Level II, pipeline autonomy: The agent runs design through testing, auditing, and deployment. Humans neither author nor review the code, stating a high-level demand and evaluating the resulting behavior instead.
Level III, demand autonomy: The agent decides what should be built, identifying work from telemetry, user behavior, security advisories, and dependency changes.
Today's AI-assisted coding sits below Level I rather than above it, and the paper's authors are explicit that Level II assumptions do not hold at scale. Level alone also underdescribes a system, because specification granularity, temporal autonomy, and oversight mode each move independently. A detailed bug report with a reproducing test constrains an agent, while a request to add multi-tenancy pushes even a nominally lower-level system into higher-autonomy territory.
Copy link to headingWhere level-skipping shows up
Level-skipping is the failure most teams actually encounter. A team operates nominally at Level I, with humans formally responsible for review and deployment, while in practice merging agent-generated changes nobody has meaningfully inspected. That team has adopted Level II practices without Level II verification, governance, or accountability, and the burden of proof has fallen at exactly the point where autonomy, duration, and risk say it should rise.
The 2× mandate study measures that drift. The share of pull requests receiving at least one human review fell 21 percentage points, from 89% to 68%. Substantive review, meaning a review carrying a human-written comment, fell from roughly 39% to 21%, while at the median, a reviewer's commented reviews held flat at about three per month, and their silent approvals roughly doubled. Review volume held up and review substance did not, which is what level-skipping looks like in production data.
Detecting it in a trial takes one real ticket at each supervision mode the agent offers, with a count of the interventions each run needs. Immediately after the merge, the reviewer records whether the code took longer to understand than a human-authored equivalent, because autonomy only pays when the resulting review stays legible.
Sentiment is the wrong measure for that test. In METR's randomized controlled trial, experienced open-source developers working on their own repositories took 19% longer to complete issues with early-2025 AI tools, having predicted a 24% speedup and still believing afterward that AI had sped them up by 20%. Self-reported satisfaction survived contact with the opposite result, so a trial collecting only developer sentiment returns the wrong answer confidently.
Copy link to headingWhat to test beyond the level
Because the harness is the product, model configurability is a property to test rather than assume. Check whether a team can select the model behind the agent, assign different models to different subagents or pipeline steps, and supply its own provider key. Harnesses increasingly expose per-step and per-subagent model assignment, which changes cost and latency profiles without changing the agent, and an agent that hard-codes a single model transfers that vendor's roadmap onto the team.
Code residency belongs in the same pass. A cloud agent sends repository contents to servers that the team doesn't control, and a local or self-hosted agent doesn't. For teams under contractual or regulatory constraints, that difference settles the shortlist before any capability question gets asked. Confirm attestation and data-retention terms during procurement rather than after the trial, since a capable agent that fails review on data handling wastes two weeks of engineering time.
Pick the highest autonomy the available review capacity absorbs, then shrink the agent's decision space with scoped tools and tightly framed tasks. That costs flexibility on open-ended work, and it buys diffs reviewers can actually read. Narrowing the decision space depends on the agent seeing the right code, which the second criterion measures.
Copy link to headingCriterion #2: Test how the AI coding agent retrieves context
Context handling is how the agent finds the right code and gets it in front of the model. Some agents query pre-computed embedding indexes semantically, others explore on demand through file reads and pattern search, and others build structured maps from parse trees and dependency graphs. The distinction shows up under evaluation as retrieval accuracy rather than as a specification number.
Window size is the wrong axis. SWE-ContextBench, a benchmark of 1,100 base tasks plus 376 related tasks drawn from real dependency and reference relationships across 51 repositories and 9 programming languages, found that accurately summarized and retrieved prior context improves resolution accuracy and cuts both runtime and token cost, with the largest gains on harder tasks. Unfiltered or incorrectly selected context delivered a limited or negative benefit. More context can make an agent worse, which means selection quality is the property under test.
Copy link to headingHow to test retrieval quality
The test is a module that the agent has never seen. Commit a new file, then ask the agent to find it and explain how it fits the change in flight. For reach, pick a change that has to propagate across several files rather than one that resolves in a single edit.
A strong result cites files that exist in the branch under review and explains why those files matter, including a file committed an hour earlier, regardless of the advertised window. A weak result reasons fluently about files that aren't there, which is the failure that produces plausible code written against interfaces that have since changed.
Both retrieval strategies carry a cost worth noting. Index-based retrieval answers fast and goes stale between reindex runs, so it degrades exactly on the recently changed code a trial cares about. On-demand exploration stays current and spends tokens and wall-clock time on every task. Pick based on whether the repository changes faster than the index refreshes. Finding the right files still doesn't follow the conventions inside them, which is the third criterion.
Copy link to headingCriterion #3: Check the coding agent against repository conventions
Repository awareness is whether the agent follows a project's conventions across many prompts, not just one. It's the criterion most likely to look fine in a demo and cost the most afterward, because its failures accumulate instead of announcing themselves.
Two tests separate an agent that reads a codebase from one that guesses at it. The reuse test asks for a utility that already exists somewhere in the repository. A repository-aware agent calls the existing helper, and a weaker one writes a second implementation under a different name. The implicit-convention test uses a rule the codebase follows without documenting, such as four-digit error codes or universally unique identifiers (UUIDs) on new records, and checks whether the agent infers the pattern from surrounding code.
Copy link to headingWhat convention files can and cannot do
Convention files are the cheapest lever available before a trial starts. AGENTS.md sits alongside README.md as agent-facing guidance covering commands, testing, project structure, code style, git workflow, and boundaries, with CLAUDE.md and a .cursor/rules/ directory serving as tool-specific equivalents. A repository without one has an improvement available for an afternoon of work.
Then test the file rather than trusting it. Favor the agent whose output changes when the convention file changes, because an agent that produces identical code with the file present and absent is not reading it, and a convention file nobody verifies becomes documentation that quietly does nothing.
Expect this criterion to degrade over a longer horizon than a demo covers. The Berkeley position paper reports the same pattern from the benchmark literature: on benchmarks built to test continuous software evolution rather than isolated tasks, frontier agents deteriorate sharply, adding features successfully while struggling to preserve correctness and architectural coherence across successive changes. The 2× mandate study points the same direction from production data, where throughput gains concentrated in repositories created after 2022 and were not statistically significant in legacy pre-2022 code. Run the trial on the code the team actually maintains, including the parts nobody wants to open. An agent that respects conventions still has to produce work a reviewer can validate, which the fourth criterion measures.
Copy link to headingCriterion #4: Test the AI coding agent's self-verification
Self-verification is whether the agent's own proof that the code works means anything. Most evaluations skip it. An agent that finds the right files and follows the house style will still hand back a green test suite that proves little about the change.
The mechanism is correlated failure. When the same agent writes the implementation and the tests, passing tests demonstrate consistency between two artifacts from one source rather than correctness against intent. Autonomous agents commonly generate implementation, tests, documentation, and rationale together, which means every artifact meant to validate the others can fail in the same direction at once. Independent verifier agents help only when they operate with genuinely independent objectives, trustworthy evaluation mechanisms, and a protocol for resolving disagreement.
Copy link to headingHow to test for correlated failure
Two tests read this criterion during a trial:
The contradicted assertion: Assign a ticket whose correct fix changes behavior an existing test asserts. A strong agent surfaces the conflict and asks which side is authoritative, and a weak one edits or deletes the assertion so the suite goes green.
The foreign suite: Run the agent's branch against tests it did not author, including integration tests it never read. Agents that report success only against self-authored suites reveal the gap immediately.
Weigh the honest cost here. Requiring independent validation adds infrastructure and slows every task, and it's the only thing that makes a passing build mean anything once the author of the code is also the author of the tests. Teams that skip it inherit the assurance problem at merge time instead, where it costs reviewer attention, which is the fifth criterion.
Copy link to headingCriterion #5: Measure how coding agent pull requests clear review
The first four criteria measure capability. This one measures delivery, and it's where promising trials stop looking promising. Track how agent output moves through the pull request process, because merged work is the only output a team can count.
The scale of the problem is measurable. In the 2× mandate study, pull request volume grew 3.1× over the pre-mandate baseline while the pool of developers doing the reviewing grew only 1.5×, so per-reviewer load doubled. Review capacity, not agent throughput, is what a trial is really testing.
The 2× mandate study quantifies the cost precisely. Comparing AI-authored and human-authored pull requests within the same author and month, controlling for change size, AI-authored pull requests post-mandate waited about 15% longer for review pickup, took about 20% longer from first human review to merge, and ran about 22% longer in total cycle time. Coding lead time, meaning first commit to pull request opened, got slightly shorter. The authoring stage sped up and the review stage absorbed the difference.
The shape of that latency matters more than its size. Review rounds and change-request counts stayed flat, and AI-authored pull requests were no less likely to receive a human review than comparable human-authored ones, so the delay is neither rework nor evasion. What lengthened is the interval between the first human review and the merge, over which AI pull requests accumulated more human comments without more revision rounds. That's more reviewer engagement, not more fixing.
Quality metrics did not move while this happened. Merge rates held steady and revert rates ran slightly lower for AI-authored changes, even though those changes were larger on average. That combination is easy to misread as reassurance, and it isn't one. Merge and revert are short-horizon proxies, and the study's authors say so directly: they miss defects, incidents, and maintainability, which are the costs that surface a quarter later rather than a week later. What the numbers rule out is an immediate quality collapse. They cannot establish whether longer-term defect, incident, or maintainability outcomes changed.
The cost that did show up caught the reviewer's attention, and the organization paid it by moving the review onto automation. Automated AI review climbed from roughly 19% to 84% of pull requests while human coverage fell. For a team reading this before its own trial, that is the pattern to watch for: the metrics on the dashboard hold while the thing that actually catches problems quietly stops happening.
Copy link to headingWhat to measure during the trial
Track four numbers against the team's existing baselines for human-authored changes: time to pull request, acceptance rate, rework rate, and queue depth. Queue depth is the leading indicator, because it moves before acceptance rate does.
Cost per task is worth tracking and worth distrusting on its own, because the pricing shape decides the bill more than any single-task figure. Rolling usage limits, usage-based credits that draw down at different rates per model, and per-seat floors on top of consumption all produce a different total at trial volume than a per-task estimate suggests. Model the cost at the volume the trial actually generates, on the plan the team would actually buy.
Teams that beat their baseline constrain the shape of what the agent submits rather than switching agents:
Pull request size caps: A diff a reviewer can finish in one sitting gets reviewed in one sitting, and the study's finding that AI pull requests run larger than human ones is the pressure these caps release.
Stacked pull requests: Splitting a full-stack change into ordered schema, API, and interface changes gives each reviewer a diff within their own competence.
No direct commits to main: Every agent task ends as a reviewable pull request, which keeps the level-skipping failure from Criterion #1 out of the default path.
Those constraints give reviewers enough context to check behavior alongside code shape, and checking behavior means running the change rather than reading it.
Copy link to headingHow Vercel supports teams running AI coding agents
A coding agent produces branches and pull requests. Everything in Criterion #4 and Criterion #5 happens after that, in the validation and review workflow that the agent hands off to. Vercel covers that half: independent validation against real builds, a running application per pull request, isolated execution for agent code, and deployment on merge without a manual pipeline step.
Copy link to heading1. Vercel Agent Code Review validates agent pull requests independently
This is the independent check Criterion #4 asked for. Most teams don't have a second system that runs the real suite against a proposed patch, so the gap gets filled by a reviewer reading a diff and trusting a green checkmark from the author of the tests.
Vercel Agent is in Beta on Pro and Enterprise plans, and its Code Review capability closes that loop. It performs multi-step reasoning to identify security vulnerabilities, logic errors, and performance issues, generates candidate patches, and runs them in secure sandboxes against the repository's real builds, tests, and linters. Only suggestions that pass those checks appear in the pull request, ready to apply in one click, so the validation is independent of the agent that authored the change.
Code Review runs when a pull request is created and when a batch of commits lands on an open one, and teams can reach it on demand by mentioning @vercel in any pull request comment to request a review, a fix, or an explanation of a failure. It also reads the same convention files Criterion #3 tests, detecting coding guidelines in a documented priority order that starts with AGENTS.md, then CLAUDE.md, then per-tool rule files including .cursor/rules/*.mdc. The file a team writes to steer its coding agent steers the reviewing agent too.
Copy link to heading2. Preview deployments extend agent review beyond the diff
Reading a large diff as text answers whether the code looks right, not whether the form submits, the loading state behaves, or the button lands where the ticket asked. Agent output widens the gap because it arrives in larger changes than human-authored work.
Vercel creates a preview deployment when a commit lands on a non-production branch or a pull request opens on GitHub, GitLab, or Bitbucket, and each deployment gets an automatically generated URL. Reviewers check the agent's change in a running instance of the application, which turns review from a reading exercise into a behavioral one.
Copy link to heading3. Git integration deploys agent-opened pull requests automatically
Cloud agents open pull requests while nobody is at a keyboard, and a pull request without a deployment waits for a human to produce one. The wait is the review-lead latency Criterion #5 measures.
Vercel deploys every branch push and merge to the production branch across GitHub, GitLab, Bitbucket, and Azure DevOps Pipelines, so an agent-created branch produces the same review artifact as a human-created one:
bash
git push origin agent/fix-checkout
The branch gets a preview deployment without further action, and deployment protection applies the team's existing approval gates to it. Merging into the production branch triggers a production deployment, with production domains repointing on success, so nobody hand-runs a pipeline to find out whether a merged agent change deploys.
Copy link to heading4. Vercel Sandbox isolates agent execution before review
Running agent-generated code to find out what it does is a security question before it's a workflow question, and local execution on developer machines is the default nobody chose deliberately.
Vercel Sandbox runs untrusted or generated code in a Firecracker microVM with its own filesystem and network, on Node.js or Python runtimes with root access, which gives agent output somewhere to execute that isn't production or a laptop. For teams building their own agent tooling, AI SDK 7 adds timeouts, durability, and sandbox support, and integrates any agent harness, including Codex and Claude Code, which keeps the execution boundary the same wherever the agent runs.
Copy link to heading5. v0 keeps generated interfaces in the same review pipeline
Interface work reviewed through a text diff has the worst feedback loop of any change type, and it's the work most likely to get merged on a screenshot instead of a review.
v0 generates code and full-stack applications from a prompt and connects to a GitHub repository, where it creates a dedicated working branch from a base branch and commits automatically as changes land. Generated work stays isolated until someone opens a pull request against the base branch, and branches other than the production branch get preview deployments. Interface iteration enters the same queue, with the same approvals, as the repo-level agent's changes.
Copy link to headingRun the trial on a real repository before committing
Reviewer capacity decides the outcome, not the ranking. A leaderboard says which agent led a benchmark last month, and says nothing about whether a given team can absorb what that agent produces, which is the constraint that determined the result in every study above. Reviewer attention is the budget all five criteria spend, and protecting it depends as much on the validation and review workflow around the agent as on the agent itself.
Alongside whichever agent a trial selects, Vercel covers that surrounding workflow:
Vercel Agent Code Review: Sandbox-validated pull request suggestions that run against the repository's real builds, tests, and linters, surfacing only what passes, and reading the same
AGENTS.mdconventions the coding agent follows. Available in Beta on Pro and Enterprise plans.Preview deployments: An automatically generated URL for every pull request, so an agent's diff gets reviewed as a running application instead of a text file.
Vercel Sandbox: Firecracker microVMs for executing agent-generated code with its own filesystem and network, away from production and developer machines.
Git integration: Automatic builds and deployments for every commit and pull request across GitHub, GitLab, Bitbucket, and Azure DevOps Pipelines, with deployment protection applying existing approval gates to agent-opened branches.
AI SDK 7: Timeouts, durability, and sandbox support for teams building their own agent tooling, with harness integration that keeps execution portable across local development, continuous integration, and production.
Set up the deployment and review side before the trial starts by creating a new project, or browse Vercel templates for a working starting point.
Copy link to headingFrequently asked questions about AI coding agents
Copy link to headingHow long should an AI coding agent trial run before the results mean anything?
Two weeks, with three to five engineers working on real tickets from their own repository rather than benchmark tasks. Score time to pull request, acceptance rate, rework rate, and queue depth against existing baselines for human-authored changes in the same codebase.
Copy link to headingDo AI coding agents make experienced developers faster?
The controlled evidence is mixed. METR's randomized trial found experienced developers 19% slower on their own mature repositories with early-2025 tools, while the 2× mandate study found a 1.46× within-developer throughput gain concentrated in newer repositories and absent in legacy code.
Copy link to headingWhy do AI coding agent pull requests take longer to merge?
Reviewer attention, not rework. Agent pull requests ran about 20% longer from first human review to merge, while review rounds stayed flat and human comments rose, meaning reviewers spent longer engaging with the change rather than sending it back for fixes.
Copy link to headingHow much human review do agent pull requests need?
Full review, the same as any contribution, plus a check in a running environment. The measured risk is silent drift: in production data, human review coverage fell from 89% to 68% of pull requests while silent approvals doubled, so review volume can hold as review substance thins.
Copy link to headingHow should a team structure an AGENTS.md file?
Treat it as project-specific guidance for agents alongside README.md, covering commands, testing, project structure, code style, git workflow, and boundaries. If a project adds nested files, document precedence so agents and reviewers don't follow conflicting guidance.