WTF are evals? How to test agents and the software they use

Agents are now users of software, doing work on people's behalf. Whether you're building the agent or the software it uses (API, CLI, MCP server, SDK), evals give you a way to measure how well that work gets done.

Agents probably missed your launch video and billboard on Highway 101, but they still go through the same funnel a human does to find and use a product: Discovery, Selection, Access, and Success.

You can build evals for any stage of this funnel. Here, we’ll focus on the final stage, Success: whether the agent completes the user’s task correctly.

That's more difficult to check than it sounds. An agent can place a “successful” order for a hot latte when the user asked for an iced one. The request worked, so the logs in your Observability stack look fine. And because LLM outputs vary between runs, one good manual test can’t tell you whether a fix worked. You need to repeat the same tasks and grade the outcomes, before and after the change.

That’s what an eval can do for you. Here’s how we build one.

Copy link to headingThe agentic testing pyramid

If a test asks “did it work?” an eval asks “how often does it work, and how well?” You give an agent a task, grade the outcome, and aggregate the results across repeated attempts.

We group agentic software tests by how many LLM calls they typically need: deterministic tests (0), one-turn evals (1), and E2E evals (2+ across a whole task). This has turned out to be a great proxy for the effort to build them, the time they take to run, and the cost of running them. Just like in the practical test pyramid, the cheap and fast tests should make up the bulk of your test suite.

To use the pyramid, start with the problem you want to catch and pick the lowest tier that can catch it. For Success, that's the top. A one-turn eval checks a single decision, like which command the agent runs first. But Success depends on the whole sequence of steps (e.g. search for cafes → filter on user criteria → place the latte order). Each one can look reasonable on its own, and the agent can still get the wrong outcome.

An E2E eval lets the agent work through the whole task in an environment, then checks whether the final result meets the user’s requirements. You’ve probably seen a few already; most popular coding benchmarks, like SWE-bench and Terminal-Bench, are E2E evals.

Copy link to headingExample

Say you own SF Beans, an API for ordering coffee in San Francisco. One of the best ways to give agents access to your service is through a CLI, so you’re shipping sfbeans.

Here’s a typical request from one of your customers:

Get me a large iced latte with whole milk, ready for pickup by 9:00, under $8 including tip. I’m in the Mission.

Here’s what sfbeans knows at 8:30am:

• Fog City Roasters: ice machine is down, so no iced drinks today (temperature)

• 24th St Beans: latte is $7.50 on the menu, but $9.27 after tax and tip (budget)

• Dolores Drip: latte is $5.50, but the earliest pickup is 9:20 (time)

• Valencia Grind: $7.42 after tax and tip, ready at 8:50 (meets all three)

Here’s what you could test at each layer of the pyramid for sfbeans:

Copy link to headingHow to build an end-to-end eval

We’ll build the coffee eval with Harbor, an open-source framework from the creators of Terminal-Bench that runs agents in containers, checks results, and records their actions.

Here are the terms we’ll use:

• Task: one problem for the agent to solve, with a starting environment and a way to check whether it succeeded

• Variant: the agent setup you’re testing, like the model, prompt, tools, memory and skills

• Trial: one attempt by one variant at one task

• Run (a job in Harbor): every variant tried on every task, several times each

• Trace (a trajectory in Harbor): everything recorded during a trial: messages, tool calls, logs, files and final output

Copy link to headingStep 1: Set a goal

Building E2E evals for everything isn’t practical. With traditional software, you can aim for 100% test coverage. With agents, the surface you’d need to cover is everything an LLM might do with your product, and even figuring out where that ends is hard (see the jagged frontier).

Start with one outcome your users care about, narrow enough to evaluate with a handful of tasks. You’ll usually find that goal in one of three places:

• Observability: which agents use your product, and where they fail

• Customers: support tickets, feedback, and churn

• Roadmap: the product hypotheses you most need to validate

For example, your goal might be to:

• Increase the success rate on a specific task

• Reduce the number of steps required to accomplish a goal

• Check whether an agent skill helps, and get feedback to improve it

• A/B test a prompt

• Explore the frontier by checking whether a use case is achievable with current LLMs, and if they fail, why

Copy link to headingExample

We looked in all three places for sfbeans:

• Observability: most agent traffic comes from Claude Code and Codex. We know because sfbeans adds a user-agent header when it detects a specific coding harness.

• Customers: the top support ticket is “my agent ordered the wrong drink”: a hot latte instead of iced, a pickup after the user’s meeting, or a total over budget once tax and tip are added.

• Roadmap: nothing urgent, so the wrong-drink problem goes first, since it drives churn.

That gives us our goal:

Copy link to headingStep 2: Define metrics

Here are the metrics we use most for E2E agent evals:

Success

• Success rate: the share of trials that pass

• Pass@k: the chance that at least one of k trials passes. Answers “can the agent do this at all?”

• Pass^k: the chance that all k trials pass. Answers “can I rely on it every time?”

Efficiency (measured on successful trials only)

• Steps: the number of agent turns needed to complete the task

• Time: how long the trial took

• Cost: how much the trial cost, in dollars

Copy link to headingExample

Our goal is to get every order right in every trial. Once we build and run the eval in Steps 3 and 4, here’s what five trials of Claude Code + Opus 5.5 (anthropic/claude-opus-5.5 on AI Gateway) on the iced latte order could look like:

What the numbers mean:

• Success rate: 3 of 5 trials passed, so 60%.

• Pass@5: assuming independent trials with a 60% success rate, 1 − 0.4^5 ≈ 99%. Give the agent 5 tries and it almost certainly gets it right once.

• Pass^5: 0.6^5 ≈ 8%. The chance the agent gets 5 orders in a row right.

• Steps: (9 + 11 + 8) / 3 ≈ 9 on successful trials. The failed trials are shorter because the agent skipped work, like quoting, which is why efficiency only counts successful trials.

• Time: ~3:50 on successful trials.

• Cost: $0.56 per successful trial, but $2.35 / 3 = $0.78 per success once you count the failed trials you paid for too.

So does Opus work? Its 99% pass@5 says yes. Its 8% pass^5 says a user who orders this every weekday gets at least one wrong coffee in 92% of weeks. Both numbers are right. They just answer different questions: pass@5 asks whether the agent can get the order right, and pass^5 asks whether it gets it right every time.

Which metric matters most depends on what a failure costs your user. Someone ordering their morning coffee doesn’t get a retry. If the agent orders a hot latte, or one that’s ready after their train leaves, they’ve paid for the wrong coffee and started their day annoyed. They need it right every morning, so sfbeans optimizes for pass^5.

Other products are more forgiving, because a failed attempt is cheap and gets caught before it matters:

Overall averages can hide a task that fails frequently, so always inspect results per task. And with only five trials, 100% means “no failures seen yet,” not “never fails.”

Copy link to headingStep 3: Design the task set

In Step 1, we set a goal for the eval: increase how often agents order the exact drink the user asked for. But an agent can’t attempt a goal. It needs a task: one concrete request, drawn from what your users actually ask for.

Pick tasks hard enough that you learn something from running them. A good set covers the common case, the edge cases that trip agents up, and at least one request where the right move is not to act.

For each task, define these parts:

• An instruction: the user’s request

• An environment: where the agent works, with the same starting state for every trial

• A verifier: a check of the final state, ideally scored per constraint

• A known-good solution (optional): a reference run that demonstrates the expected behavior and passes the verifier

You can make every aspect of that environment as fake or as realistic as you need, depending on your goals. The OpenPipe team has a nice writeup on how they defined the environment for an email research agent.

Copy link to headingExample

We wrote four tasks, each one a Harbor task folder in tasks/.

• Common case: iced-latte-mission, the order from earlier

• Edge case: usual-unavailable, where the user’s usual cafe is out of their drink

• Edge case: team-order, with 6 drinks, $45 total, and one pickup spot

• Don’t act: impossible-request, where nothing fits, so the right move is to tell the user

impossible-request has one extra check. Its verifier reads a command log the agent can’t write to, to confirm the agent searched before giving up. We learned we needed that the hard way (see Step 5).

Here’s iced-latte-mission:

├── instruction.md        # the initial user prompt
├── task.toml             # task config: timeouts, verifier, artifacts
├── environment/          # Docker build context
│   ├── Dockerfile        # sfbeans + seeded cafes, clock frozen at 8:30am
│   ├── sfbeans           # the CLI under test
│   └── cafes.json        # seeded cafes, menus and pickup times
├── solution/
│   └── solve.sh          # a known-good answer, proves the task is solvable
└── tests/
    ├── test.sh           # the verifier
    └── check_orders.py

The instruction gives the agent the user’s request and names the sfbeans CLI. In production, agents find the tool through a skill, docs, or an MCP server. Naming it here skips Discovery and Selection, keeping the eval focused on Success.

We also specify that it’s 8:30am. Only sfbeans uses our frozen clock; the sandbox’s system clock still shows the real time:

Use the sfbeans CLI to get me a large iced latte with whole milk, ready for
pickup by 9:00am today, under $8 including tip. I'm in the Mission.
It's 8:30am now.

The environment is part real, part fake. The agent uses the real sfbeans CLI, but against seeded cafes and a frozen clock, so every trial starts on the same morning:

1
FROM python:3.12-slim
2
COPY sfbeans /usr/local/bin/sfbeans
3
COPY cafes.json /opt/sfbeans/cafes.json
4
ENV SFBEANS_DATA=/opt/sfbeans/cafes.json \
5
    SFBEANS_ORDERS=/opt/sfbeans/orders.json \
6
    SFBEANS_NOW="08:30"

To keep the demo simple, cafes.json lives inside the agent’s environment. In a real eval, protect the seed data and verifier code, and make sure the agent can update order records only through the CLI. Agents can attempt to reward hack if it helps them pass, for example by editing the inventory or writing orders.json directly.

For this coffee order, the verifier checks the final order against the user’s requirements. Harbor runs test.sh after the agent finishes and reads the scores it writes to /logs/verifier/reward.json:

python3 /tests/check_orders.py /opt/sfbeans/orders.json > /logs/verifier/reward.json

The actual check lives in check_orders.py. It scores each constraint separately, then sets reward to 1 only if all of them pass. Harbor uses reward as the trial’s main score:

1
import json, os, sys
2

3
path = sys.argv[1]
4
orders = json.load(open(path)) if os.path.exists(path) else []
5
o = orders[0] if len(orders) == 1 else {}
6

7
reward = {
8
    "one_order": int(len(orders) == 1),
9
    "right_drink": int(o.get("drink") == "latte" and o.get("size") == "l"
10
                       and o.get("temp") == "iced" and o.get("milk") == "whole"),
11
    "on_time": int(o.get("ready_at", "99:99") <= "09:00"),
12
    "on_budget": int(o.get("total", 99) <= 8.00),
13
}
14
reward["reward"] = int(all(reward.values()))  # 1 only if every constraint holds
15
print(json.dumps(reward))

The solution is a known-good answer. Run it before any real agent to prove the task works:

sfbeans quote cafe_val07 --drink drink_lat_l --size l --temp iced --milk whole --tip 15%
sfbeans order cafe_val07 --drink drink_lat_l --size l --temp iced --milk whole --pickup 08:50 --tip 15%

Finally, task.toml ties the task together. It sets the timeouts, runs the verifier in the agent’s sandbox (where orders.json lives), and saves the orders as an artifact so we can inspect them in Step 5:

1
artifacts = ["/opt/sfbeans/orders.json"]
2

3
[task]
4
name = "sfbeans/iced-latte-mission"
5

6
[agent]
7
timeout_sec = 600
8

9
[verifier]
10
timeout_sec = 60
11
environment_mode = "shared"  # verify in the agent's sandbox, where orders.json lives

Copy link to headingStep 4: Run the eval

If you share the universal distaste for slow CI, you’ll hate waiting for evals. Every trial waits on an agent making LLM calls, step after step, and a run repeats that for every variant and task, several times each.

Harbor’s Vercel integration, which we built upstream, speeds that up a lot. It runs trials concurrently, each in its own Vercel Sandbox, and caches built task environments so they aren’t rebuilt for every trial. Routing model calls through AI Gateway gives you one key and one budget across model providers.

Copy link to headingExample

Two agents, Claude Code and Codex, with two models each, make four variants.

1
job_name: sfbeans-baseline
2
n_attempts: 5            # trials per task, per variant
3
n_concurrent_trials: 10  # sandboxes running at once (the Hobby max)
4

5
environment:
6
  type: vercel
7

8
agents:
9
  - name: claude-code
10
    model_name: anthropic/claude-opus-5.5
11
    env:
12
      ANTHROPIC_AUTH_TOKEN: ${VERCEL_OIDC_TOKEN}  # visible to the agent, fine for a demo
13
      ANTHROPIC_API_KEY: ""
14
      ANTHROPIC_BASE_URL: https://ai-gateway.vercel.sh/claude-code
15
  - name: codex
16
    model_name: openai/gpt-6-astra
17
    env:
18
      OPENAI_API_KEY: ${VERCEL_OIDC_TOKEN}
19
      OPENAI_BASE_URL: https://ai-gateway.vercel.sh/codex/v1
20
  # Same pattern for anthropic/claude-sonnet-5.5 and openai/gpt-6.1-sol
21

22
datasets:
23
  - path: tasks/

Then install Harbor, connect a Vercel project, and run the job:

uv tool install --python 3.12 'harbor[vercel]==0.24.0' --with 'vercel==0.10.0'  # versions we tested

vercel login      # authenticate
vercel link       # pick the team + project
vercel env pull   # writes VERCEL_OIDC_TOKEN to .env.local (valid 12h)

harbor run -c job.yaml   # 4 tasks × 4 variants × 5 attempts = 80 trials, 10 at a time
harbor view jobs         # browse results and trajectories

This run fits on the Hobby plan: 10 concurrent sandboxes is the Hobby limit, and each trial is well under its 45-minute session cap. On Pro, you can raise n_concurrent_trials as high as 10,000 (Sandbox limits).

To keep the example simple, we pass the OIDC token into each sandbox as an env var, where the agent can read it. Beyond a demo, use short-lived tokens, and use Harbor’s Vercel integration to set Sandbox firewall rules that lock each trial down to only the hosts it needs, like AI Gateway.

Internally, we go a step further and use credentials brokering to attach credentials to outgoing requests, keeping API keys outside the agent’s environment.

Copy link to headingStep 5: Analyze traces

Now for the meaty part: trace-diving. Scores tell you how often your agents fail; traces tell you why. Read the failures, and some passes too, since a pass only means the verifier was satisfied.

Copy link to headingExample

The run from Step 4 leaves 80 trial folders in jobs/sfbeans-baseline/. Each has the trial’s result, the agent’s trajectory, the verifier’s scores, and, because task.toml saves it as an artifact, the order the agent placed:

├── result.json                     # results across all variants and tasks
├── iced-latte-mission__Ab3dE7x/    # one trial, with a generated suffix
│   ├── config.json                 # which agent, model and task
│   ├── result.json                 # trial scores, cost and timing
│   ├── trial.log                   # runner logs, useful if the trial breaks
│   ├── agent/
│   │   └── trajectory.json         # what the agent saw, said and did
│   ├── verifier/
│   │   ├── reward.json             # did the order meet every constraint?
│   │   └── test-stdout.txt         # verifier output
│   └── artifacts/
│       └── opt/sfbeans/orders.json # the order the agent actually placed
└── …                               # the other 79 trials

Run harbor view jobs to browse the results. Here’s what our four variants could report (numbers are illustrative).

For each variant, we calculate success rate and estimate pass^5 separately for each task, then average each metric across the four tasks. Each task counts equally.

That’s our baseline. On pass^5, the metric we chose in Step 2, even the best variant is far from reliable. But the score doesn’t tell us what to fix. For that, we have to open the failed trials.

Take trial 2 from Step 2. Claude Code + Opus 5.5 ordered at 24th St Beans for $9.27. The verifier tells us which constraint it broke:

{
  "one_order": 1,
  "right_drink": 1,
  "on_time": 1,
  "on_budget": 0,
  "reward": 0
}

Right drink, on time, over budget, and the saved orders.json confirms the $9.27 total. Now we read agent/trajectory.json. The agent searches for lattes in the Mission, sees $7.50 on the menu, and goes straight to order. No quote. It treated the menu price as the final price.

Trial 1, from the same variant, quotes three cafes, rules out 24th St Beans, and orders at Valencia Grind for $7.42. So the CLI can get the order right, but the agent doesn’t reliably check the total before spending the user’s money.

We then look for that pattern across other failed trials, and if it keeps showing up, we have something specific to fix: make the total harder to miss, or add a --max-total flag so order refuses anything over budget. That’s a much more useful conclusion than “Opus scored 85%.”

When we first ran this demo, the traces also exposed problems with what the eval was measuring:

• Our first run scored 0% on every variant. The agents didn’t know sfbeans existed, so they never tried it (obviously). That’s the funnel’s Discovery stage failing in our own eval. Naming the tool in the instruction fixed it, and it’s a good reminder that being discoverable is part of your product. In the wild, that’s the job of your skill, docs, MCP server and AEO.

• Codex kept looking for sfbeans as a connected tool and never tried running it in the shell. That’s an Access problem, not a Success one.

• Codex passed impossible-request almost every time, but when we read the traces, 9 out of 10 passes were the agent giving up without looking. The verifier only checked that no order was placed, so we tightened it to also check that the agent searched first. Without reading the traces, we’d have counted those as wins.

Copy link to headingAutomated trace analysis

For larger runs, inspecting every trial by hand gets slow, so we automate it.

Up to 40 reader agents running fast models like deepseek/deepseek-v4.1-flash read trial transcripts in parallel and summarize their findings. A smarter coordinator agent reviews those summaries, sends readers back for more context when needed, and writes the final report. An analysis.md file tells the coordinator what to look for, so we can reuse the workflow across runs.

Model calls go through AI Gateway. Trial artifacts stay in Vercel Drives during analysis, and traces go to Vercel Blob for long-term storage.

Copy link to headingIterate to greatness

Your first eval run gives you a baseline. Now act on it.

Pick the fix the traces point to, like the --max-total flag, rerun the same job, and compare against the baseline. Check whether it reduces the targeted failures without introducing new ones, then use those results to decide whether the change is ready to ship.

Once you’ve built an eval, it keeps paying off for a lot of different things. Rerun it on every release to catch regressions; test prompt, skill, and model changes before shipping; learn how agents use your product; or build auto-optimization loops on top of it.

Expect to add harder tasks over time. As models and your product improve, evals saturate, meaning agents start passing every task. That’s great for regression detection, but it can't tell you what to improve next.

Evals are the infrastructure you use to make your product better, again and again. Vercel has the pieces you need to build them in one place: AI Gateway for model calls across providers, Sandbox to run trials at scale, and Drives and Blob to keep every trace.

Ready to run your first eval? Paste this prompt into your coding agent, and it’ll build and run one for your product on Vercel with you: