Harbor and Terminal-Bench

Evaluating a coding agent means running it against hundreds of benchmark tasks, each in a clean environment, and scoring the results. An evaluation harness automates that loop: it prepares each task's environment, runs the agent, and scores the outcome. Harbor is the open source harness built by the Terminal-Bench team, and one integration covers its whole registry: any of the 300 or more datasets on the Harbor Hub, including Terminal-Bench, SWE-bench, tau3-bench, and OSWorld, runs every trial in its own isolated Vercel Sandbox microVM instead of a local Docker container. Trials run in the cloud, so you can run many more in parallel than on your local machine.

Before you begin, make sure you have:

• uv or pip to install Harbor
• A Vercel access token, or a Git repository linked to a Vercel project with vercel link
• An AI Gateway API key: one key gives your agent access to hundreds of models from multiple providers, so you can benchmark different models without new credentials. A single provider's key, such as ANTHROPIC_API_KEY, works too. The built-in oracle agent replays each task's reference solution and needs no model key at all.

Sandbox time that Harbor uses is billed as normal Vercel Sandbox usage. Trial sandboxes are deleted when the trial ends and leave no snapshots behind. Harbor's one-time build caches (the builder snapshot behind image builds and Compose hosts) persist on the harbor-sandbox project and are billed as snapshot storage.

1. Install Harbor with the Vercel extra

   Terminal

   uv tool install 'harbor[vercel]'

   The vercel extra installs the Vercel Python SDK. The square brackets are typed literally, and the quotes keep your shell from treating them as glob characters. You need Harbor 0.22.0 or later. If an older Harbor is already installed, see Troubleshooting to upgrade.

2. Authenticate with Vercel

   If you work in a repository linked to a Vercel project, export a development token for that project:

   Terminal

   vercel link
   export VERCEL_OIDC_TOKEN="$(vercel project token)"

   Harbor then authenticates as your linked Vercel project. The token is valid for 12 hours, so run the export again when it expires.

   Without a linked project, set an access token instead. Harbor resolves your team, creates a project named harbor-sandbox, and scopes everything to it:

   Terminal

   export VERCEL_TOKEN="<your-token>"

   If your token has access to more than one team, also set VERCEL_TEAM_ID. See Sandbox authentication for both methods.

3. Run a smoke test

   Run the single hello-world task with the built-in oracle agent, which replays each task's known-correct reference solution instead of calling a model, so it verifies your setup without needing a model key:

   Terminal

   harbor run -t hello-world/hello-world -a oracle -e vercel

   The first run builds the task's image into Vercel Container Registry and finishes with a reward of 1.0. Repeat runs boot the cached image directly.

4. Run Terminal-Bench

   Evaluate an agent and model against Terminal-Bench 2.1 (89 tasks). The gateway-aware fx agent reads AI_GATEWAY_API_KEY, so one AI Gateway key reaches models from every provider:

   Terminal

   export AI_GATEWAY_API_KEY="<your-key>"
    
   harbor run -d terminal-bench/terminal-bench-2-1 \
     --agent fx \
     --model vercel_ai_gateway/anthropic/claude-fable-5 \
     --env vercel \
     --n-concurrent 8

   The example's concurrency of 8 fits every plan's concurrency limit, including Hobby's 10 concurrent sandboxes. Hobby also caps each session at 45 minutes, below Harbor's default sandbox lifetime of 24 hours, so on Hobby lower the lifetime with the sandbox_lifetime_seconds option shown in Configuration. On paid plans, raise --n-concurrent toward your model provider's rate limits. The same command runs any dataset registered on the Harbor Hub.

5. View results

   Terminal

   harbor view jobs

   Harbor writes results to the jobs directory by default. Pass a different path if you set --jobs-dir. This starts a local web viewer and prints its URL. Open it in your browser and stop the server with Ctrl+C when done. On disk, each trial's record also includes a vercel-env-setup JSON file with per-phase setup timings and the image or snapshot the sandbox booted from.

A trial moves through three stages. Harbor turns the task's environment definition into a sandbox image, boots a fresh sandbox from that image for the agent to work in, and applies the task's network rules to the sandbox firewall. Each item below covers one stage of that flow.

• Native image boot: Harbor builds a task's image once and stores it in Vercel Container Registry. Every trial of that task boots its sandbox directly from the stored image. The image is keyed by the task's content, so editing a task rebuilds its image and unchanged tasks reuse theirs. Pass the task_image option to boot from an image already in the registry.
• Docker Compose tasks: Tasks that need several containers, such as an app plus its database, run their Compose file inside the sandbox. Because installing Docker takes 30 to 90 seconds, Harbor does it once, saves the sandbox as a snapshot, and starts later trials from that snapshot. Reusing the snapshot only makes trials start faster. It cannot change their results.
• Network modes: A task's network policy maps to the sandbox firewall: public, no network, or a hostname allowlist with wildcard support, enforced outside the VM. Compose tasks support the public and no-network modes.
• Credential injection: The credential_injection option attaches secrets to matching outbound requests at the sandbox firewall, so API keys never enter the sandbox. Available for single-container tasks.
• Resource requests: A task's CPU and memory requests from task.toml are applied to the sandbox, and --override-cpus and --override-memory work as with other providers. GPU tasks are not supported.

Harbor accepts extra environment options with --ek key=value (short for environment keyword argument). Values are JSON-parsed, so nested options are passed as one shell-quoted argument, for example --ek 'credential_injection={"api.example.com": {"headers": {"Authorization": "Bearer token"}}}':

The Vercel CLI reads the VERCEL_TOKEN environment variable, so a placeholder value like <your-token> left exported from an earlier step makes vercel link reject it. Run unset VERCEL_TOKEN, then retry the link and pull.

Harbor cannot pick a team on your behalf. Set VERCEL_TEAM_ID to the team that should own the harbor-sandbox project.

Harbor 0.22.0 does not read .env.local on its own, so credentials from vercel env pull stay invisible to it. Export a token as shown in step 2, or point Harbor at the file with harbor run --env-file .env.local.

Your installed Harbor predates the integration. Releases before 0.22.0 do not contain the Vercel environment. Upgrade with uv tool uninstall harbor followed by uv tool install 'harbor[vercel]'.

Tasks with a docker-compose.yaml support the public and no-network modes but not hostname allowlists. Harbor fails the trial at validation time with an explanatory error rather than running with a weaker policy.