Copy link to headingIntroduction
This guide shows you how to run Terminal-Bench, or any other benchmark registered on the Harbor Hub, with every trial executing in its own isolated Vercel Sandbox microVM instead of a local Docker container. You will install Harbor, authenticate with one Vercel token, run a smoke test, run Terminal-Bench, and compare models through AI Gateway by changing a single flag.
Copy link to headingOverview
An evaluation harness automates benchmark runs: it prepares each task's environment, runs your agent, and scores the outcome. Harbor is the open source harness built by the Terminal-Bench team. Its hub registers 274 datasets, including Terminal-Bench, SWE-bench, tau3-bench, and OSWorld, and all of them run through the same harbor run command. Passing --env vercel moves each trial into a Vercel Sandbox microVM. Trials run in the cloud, so you can run many more in parallel than on your local machine.
Paired with AI Gateway, one AI_GATEWAY_API_KEY reaches hundreds of models from multiple providers, so benchmarking another model is the same command with a different --model flag.
Copy link to headingPrerequisites
- uv or pip to install Harbor 0.22.0 or later
- A Vercel account and access token, or a repository linked to a Vercel project with
vercel link - An AI Gateway API key: one key gives your agent access to hundreds of models from multiple providers, so you can benchmark different models without new credentials. A single provider's key, such as
ANTHROPIC_API_KEY, works too. The built-inoracleagent replays each task's reference solution and needs no model key at all.
Sandbox time that Harbor uses is billed as normal Vercel Sandbox usage. Trial sandboxes are deleted when the trial ends and leave no snapshots behind. Harbor's one-time build caches (the builder snapshot behind image builds and Compose hosts) persist on the harbor-sandbox project and are billed as snapshot storage.
Copy link to headingHow it works
- Harbor builds each task's image once, stores it in Vercel Container Registry, and boots every sandbox straight from it. Editing a task rebuilds its image; unchanged tasks reuse theirs.
- Multi-container Docker Compose tasks work too: the first run installs Docker in the sandbox and snapshots it, and later trials boot from that snapshot. Reusing the snapshot only makes trials start faster. It cannot change their results.
- A task's network rules carry over to the sandbox firewall: public, no network, or a hostname allowlist, enforced outside the VM. Optional credential injection attaches secrets to matching outbound requests at that firewall, so credentials never enter the sandbox.
Copy link to headingSteps
Copy link to heading1. Install Harbor with the Vercel extra
The square brackets are typed literally: [vercel] is Python packaging extras syntax and installs the Vercel Python SDK alongside Harbor. The quotes keep zsh from treating the brackets as glob characters. You need Harbor 0.22.0 or later. If an older Harbor is already installed, see Troubleshooting to upgrade.
Copy link to heading2. Authenticate with Vercel
If you work in a repository linked to a Vercel project, export a development token for that project:
Harbor then authenticates as your linked Vercel project. The token is valid for 12 hours, so run the export again when it expires.
Without a linked project, set an access token instead. Harbor resolves your team, creates a project named harbor-sandbox, and scopes everything to it:
If your token has access to more than one team, also set VERCEL_TEAM_ID. See Sandbox authentication for both methods.
Copy link to heading3. Run a smoke test
Run the single hello-world task with the built-in oracle agent, which replays each task's known-correct reference solution instead of calling a model, so it verifies your setup without needing a model key:
The first run builds the task's image into Vercel Container Registry and finishes with a reward of 1.0. Repeat runs boot the cached image directly.
Copy link to heading4. Run Terminal-Bench
The concurrency of 8 fits every plan's concurrency limit, including Hobby's 10 concurrent sandboxes. Hobby also caps each session at 45 minutes, below Harbor's default sandbox lifetime of 24 hours, so on Hobby lower the lifetime with the sandbox_lifetime_seconds option shown in Configuration. On paid plans, raise --n-concurrent toward your model provider's rate limits.
Copy link to heading5. Compare models
Rerun the same command with a different --model, for example vercel_ai_gateway/openai/gpt-5.6-luna. Nothing else changes: same tasks, same sandboxes, same key, and spend for every model shows up in one place.
Copy link to heading6. View results
Harbor writes results to the jobs directory by default. Pass a different path if you set --jobs-dir. This starts a local web viewer and prints its URL (for example http://127.0.0.1:8081). Open it in your browser, and stop the server with Ctrl+C when done. On disk, each trial's record also includes a vercel-env-setup JSON file with per-phase setup timings and the image or snapshot the sandbox booted from.
Copy link to headingConfiguration
Harbor accepts extra environment options with --ek key=value (short for environment keyword argument):
| Option | What it does | Default |
|---|---|---|
task_image | Boot directly from an existing VCR image instead of building one | None |
project_name | Vercel project that owns sandboxes, images, and snapshot caches | harbor-sandbox |
sandbox_lifetime_seconds | The sandbox VM's own lifetime deadline, separate from agent timeouts | 86400 (24 hours) |
credential_injection | Per-host firewall rules that attach headers to outbound requests | None |
--ek parses JSON values, so nested options are passed as one shell-quoted argument, for example --ek 'credential_injection={"api.example.com": {"headers": {"Authorization": "Bearer token"}}}'.
Copy link to headingTroubleshooting
Copy link to headingHarbor asks for authentication after you pulled environment variables
Harbor 0.22.0 does not read .env.local on its own, so credentials from vercel env pull stay invisible to it. Export a token as shown in step 2, or point Harbor at the file with harbor run --env-file .env.local.
Copy link to headingvercel is not a valid environment type
Your installed Harbor predates the integration. Releases before 0.22.0 do not contain the Vercel environment, and installs on those versions warn that the vercel extra does not exist. Upgrade with uv tool uninstall harbor followed by uv tool install 'harbor[vercel]'.
Copy link to headingvercel link fails with "You defined --token, but its contents are invalid"
The Vercel CLI reads the VERCEL_TOKEN environment variable, so if a placeholder value like <your-token> is still exported from an earlier step, vercel link rejects it. Run unset VERCEL_TOKEN, then retry the link.
Copy link to headingVercelScopeError: token has access to multiple teams
Harbor cannot pick a team on your behalf. Set VERCEL_TEAM_ID to the team that should own the harbor-sandbox project.
Copy link to headingAgentAuthenticationError with the claude-code agent
For anthropic/... models, Harbor reads the key from ANTHROPIC_API_KEY only. Setting ANTHROPIC_AUTH_TOKEN alone is not picked up. For gateway routing, either use a gateway-aware agent with vercel_ai_gateway/... model names and AI_GATEWAY_API_KEY, or set ANTHROPIC_API_KEY together with ANTHROPIC_BASE_URL.
Copy link to headingA trial fails with AgentSafetyRefusalError
Some models decline security-adjacent benchmark tasks; the refusal comes from the model provider's safety filters, not from Harbor or the sandbox. Harbor does not retry these by default, so the trial fails cleanly. Rerunning with a different --model is often enough.
Copy link to headingNetwork allowlists on Compose tasks
Tasks with a docker-compose.yaml support the public and no-network modes but not hostname allowlists. Harbor fails the trial at validation time with an explanatory error rather than running with a weaker policy.
Copy link to headingConcurrency on Hobby
The Hobby plan runs up to 10 concurrent sandboxes, so keep --n-concurrent at or below 10, and its 45-minute session cap requires lowering sandbox_lifetime_seconds below Harbor's 24-hour default. See Sandbox pricing and limits.
Copy link to heading
Copy link to headingFAQ
What am I billed for?
Sandbox compute during trials, billed as normal Sandbox usage: active CPU, provisioned memory, creations, and network egress (data the sandbox downloads is free). Task images Harbor builds are stored in Vercel Container Registry and billed as image storage, and Harbor's build caches are billed as snapshot storage. Model calls through AI Gateway are billed at provider token prices with no markup.
Do trial sandboxes stick around after a run?
No. Trial sandboxes are deleted when the trial ends and leave no snapshots behind. What persists is Harbor's build cache: a stopped builder sandbox whose snapshot backs image builds and Compose hosts, because reusing it is what makes later runs start faster.
Can the agent inside a trial reach the internet?
Only what the task's network policy allows: public, no network, or a hostname allowlist, enforced by the sandbox firewall outside the VM. The agent cannot loosen the policy from inside.
How do I give a trial an API key without exposing it to the agent?
Use the credential_injection option. The firewall attaches the secret to matching outbound requests, so the agent can use the credential without ever being able to read it. Available for single-container tasks.
Do GPU benchmark tasks work?
Not yet. The Vercel environment supports CPU and memory requests from a task's task.toml, but not GPU tasks such as Terminal-Bench 3's H100 tasks.
Which agents can I use?
Any agent registered in Harbor, including claude-code, codex, and the no-model oracle verifier. For comparing models across providers through one key, use a gateway-aware agent such as fx with vercel_ai_gateway/... model names.