--- title: Run Terminal-Bench and any Harbor benchmark on Vercel Sandbox description: Run Terminal-Bench, SWE-bench, and any Harbor benchmark on Vercel Sandbox. Each trial executes in an isolated Firecracker microVM, with AI Gateway routing one API key to hundreds of models. url: /kb/guide/run-terminal-bench-harbor-benchmarks-vercel-sandbox canonical_url: "https://vercel.com/kb/guide/run-terminal-bench-harbor-benchmarks-vercel-sandbox" published: 2026-09-12 last_updated: 2026-09-12 authors: Elisabeth Rülke, George Fahmy related: - /docs/ai-gateway - /docs/sandbox/pricing - /docs/sandbox/concepts/authentication install_vercel_plugin: npx plugins add vercel/vercel-plugin --- ## Introduction This guide shows you how to run Terminal-Bench, or any other benchmark registered on the Harbor Hub, with every trial executing in its own isolated Vercel Sandbox microVM instead of a local Docker container. You will install Harbor, authenticate with one Vercel token, run a smoke test, run Terminal-Bench, and compare models through AI Gateway by changing a single flag. ## Overview An evaluation harness automates benchmark runs: it prepares each task's environment, runs your agent, and scores the outcome. [Harbor](https://harborframework.com) is the open source harness built by the [Terminal-Bench](https://www.tbench.ai/) team. Its hub registers 274 datasets, including Terminal-Bench, SWE-bench, tau3-bench, and OSWorld, and all of them run through the same `harbor run` command. Passing `--env vercel` moves each trial into a Vercel Sandbox microVM. Trials run in the cloud, so you can run many more in parallel than on your local machine. Paired with [AI Gateway](https://vercel.com/docs/ai-gateway), one `AI_GATEWAY_API_KEY` reaches hundreds of models from multiple providers, so benchmarking another model is the same command with a different `--model` flag. ## Prerequisites - [uv](https://docs.astral.sh/uv/) or pip to install Harbor 0.22.0 or later - A Vercel account and access token, or a repository linked to a Vercel project with `vercel link` - An [AI Gateway API key](https://vercel.com/docs/ai-gateway): one key gives your agent access to hundreds of models from multiple providers, so you can benchmark different models without new credentials. A single provider's key, such as `ANTHROPIC_API_KEY`, works too. The built-in `oracle` agent replays each task's reference solution and needs no model key at all. Sandbox time that Harbor uses is billed as normal [Vercel Sandbox usage](https://vercel.com/docs/sandbox/pricing). Trial sandboxes are deleted when the trial ends and leave no snapshots behind. Harbor's one-time build caches (the builder snapshot behind image builds and Compose hosts) persist on the `harbor-sandbox` project and are billed as snapshot storage. ## How it works - Harbor builds each task's image once, stores it in Vercel Container Registry, and boots every sandbox straight from it. Editing a task rebuilds its image; unchanged tasks reuse theirs. - Multi-container Docker Compose tasks work too: the first run installs Docker in the sandbox and snapshots it, and later trials boot from that snapshot. Reusing the snapshot only makes trials start faster. It cannot change their results. - A task's network rules carry over to the sandbox firewall: public, no network, or a hostname allowlist, enforced outside the VM. Credential injection, which no other Harbor sandbox provider currently offers, attaches secrets to matching outbound requests at that firewall, so credentials never enter the sandbox. ## Steps ### 1\. Install Harbor with the Vercel extra ```bash uv tool install 'harbor[vercel]' ``` The square brackets are typed literally: `[vercel]` is Python packaging extras syntax and installs the Vercel Python SDK alongside Harbor. The quotes keep zsh from treating the brackets as glob characters. You need Harbor 0.22.0 or later. If an older Harbor is already installed, see Troubleshooting to upgrade. ### 2\. Authenticate with Vercel If you work in a repository linked to a Vercel project, export a development token for that project: ```bash vercel link export VERCEL_OIDC_TOKEN="$(vercel project token)" ``` Harbor then authenticates as your linked Vercel project. The token is valid for 12 hours, so run the export again when it expires. Without a linked project, set an access token instead. Harbor resolves your team, creates a project named `harbor-sandbox`, and scopes everything to it: ```bash export VERCEL_TOKEN="" ``` If your token has access to more than one team, also set `VERCEL_TEAM_ID`. See [Sandbox authentication](https://vercel.com/docs/sandbox/concepts/authentication) for both methods. ### 3\. Run a smoke test Run the single `hello-world` task with the built-in `oracle` agent, which replays each task's known-correct reference solution instead of calling a model, so it verifies your setup without needing a model key: ```bash harbor run -t hello-world/hello-world -a oracle -e vercel ``` The first run builds the task's image into Vercel Container Registry and finishes with a reward of 1.0. Repeat runs boot the cached image directly. ### 4\. Run Terminal-Bench ```bash export AI_GATEWAY_API_KEY="" harbor run -d terminal-bench/terminal-bench-2-1 \ --agent fx \ --model vercel_ai_gateway/anthropic/claude-fable-5 \ --env vercel \ --n-concurrent 8 ``` The concurrency of 8 fits every plan's concurrency limit, including Hobby's 10 concurrent sandboxes. Hobby also caps each session at 45 minutes, below Harbor's default sandbox lifetime of 24 hours, so on Hobby lower the lifetime with the `sandbox_lifetime_seconds` option shown in Configuration. On paid plans, raise `--n-concurrent` toward your model provider's rate limits. ### 5\. Compare models Rerun the same command with a different `--model`, for example `vercel_ai_gateway/openai/gpt-5.6-luna`. Nothing else changes: same tasks, same sandboxes, same key, and spend for every model shows up in one place. ### 6\. View results ```bash harbor view jobs ``` Harbor writes results to the `jobs` directory by default. Pass a different path if you set `--jobs-dir`. This starts a local web viewer and prints its URL (for example `http://127.0.0.1:8081`). Open it in your browser, and stop the server with Ctrl+C when done. On disk, each trial's record also includes a `vercel-env-setup` JSON file with per-phase setup timings and the image or snapshot the sandbox booted from. ## Configuration Harbor accepts extra environment options with `--ek key=value` (short for environment keyword argument): | Option | What it does | Default | | -------------------------- | -------------------------------------------------------------------- | ---------------- | | `task_image` | Boot directly from an existing VCR image instead of building one | None | | `project_name` | Vercel project that owns sandboxes, images, and snapshot caches | `harbor-sandbox` | | `sandbox_lifetime_seconds` | The sandbox VM's own lifetime deadline, separate from agent timeouts | 86400 (24 hours) | | `credential_injection` | Per-host firewall rules that attach headers to outbound requests | None | `--ek` parses JSON values, so nested options are passed as one shell-quoted argument, for example `--ek 'credential_injection={"api.example.com": {"headers": {"Authorization": "Bearer token"}}}'`. ## Troubleshooting ### Harbor asks for authentication after you pulled environment variables Harbor 0.22.0 does not read `.env.local` on its own, so credentials from `vercel env pull` stay invisible to it. Export a token as shown in step 2, or point Harbor at the file with `harbor run --env-file .env.local`. ### `vercel` is not a valid environment type Your installed Harbor predates the integration. Releases before 0.22.0 do not contain the Vercel environment, and installs on those versions warn that the `vercel` extra does not exist. Upgrade with `uv tool uninstall harbor` followed by `uv tool install 'harbor[vercel]'`. ### vercel link fails with "You defined --token, but its contents are invalid" The Vercel CLI reads the `VERCEL_TOKEN` environment variable, so if a placeholder value like `` is still exported from an earlier step, `vercel link` rejects it. Run `unset VERCEL_TOKEN`, then retry the link. ### VercelScopeError: token has access to multiple teams Harbor cannot pick a team on your behalf. Set `VERCEL_TEAM_ID` to the team that should own the `harbor-sandbox` project. ### AgentAuthenticationError with the claude-code agent For `anthropic/...` models, Harbor reads the key from `ANTHROPIC_API_KEY` only. Setting `ANTHROPIC_AUTH_TOKEN` alone is not picked up. For gateway routing, either use a gateway-aware agent with `vercel_ai_gateway/...` model names and `AI_GATEWAY_API_KEY`, or set `ANTHROPIC_API_KEY` together with `ANTHROPIC_BASE_URL`. ### A trial fails with AgentSafetyRefusalError Some models decline security-adjacent benchmark tasks; the refusal comes from the model provider's safety filters, not from Harbor or the sandbox. Harbor does not retry these by default, so the trial fails cleanly. Rerunning with a different `--model` is often enough. ### Network allowlists on Compose tasks Tasks with a `docker-compose.yaml` support the public and no-network modes but not hostname allowlists. Harbor fails the trial at validation time with an explanatory error rather than running with a weaker policy. ### Concurrency on Hobby The Hobby plan runs up to 10 concurrent sandboxes, so keep `--n-concurrent` at or below 10, and its 45-minute session cap requires lowering `sandbox_lifetime_seconds` below Harbor's 24-hour default. See [Sandbox pricing and limits](https://vercel.com/docs/sandbox/pricing).