Harbor with AI Gateway

Harbor runs benchmark tasks in isolated sandboxes and records the results. A coding-agent harness, such as Claude Code or Codex, performs each task. Harbor uses an adapter to install and launch the selected harness, then pass its model, endpoint, and credentials into the sandbox.

To route a harness through AI Gateway, configure the Harbor adapter for that harness. Each adapter forwards different credentials, endpoints, and model-name formats.

To run a benchmark locally, you need:

• Python 3.12 or later and uv
• Docker running on your machine
• An AI Gateway API key

The examples below use Harbor 0.22.0.

Install or update Harbor:

uv tool install --upgrade harbor

Export your AI Gateway API key for the examples below:

export AI_GATEWAY_API_KEY="your-ai-gateway-api-key"

Select the coding-agent harness you want to evaluate with --agent. Harbor runs the benchmark and uses the corresponding adapter to configure the harness inside the task sandbox. The harness solves the task; the adapter only connects Harbor to that harness.

These Harbor adapters expose settings you can use to connect to AI Gateway. Set the credential variable to your AI Gateway API key and configure the endpoint shown:

For the OpenAI-compatible paths above, model arguments may include a transport prefix in addition to AI Gateway's creator/model-name slug. OpenCode, OpenHands, Pi, Qwen Code, and Vibe can use openai/anthropic/claude-opus-5 with these settings. Aider needs openai/openai/anthropic/claude-opus-5: Harbor removes the first openai/, and Aider's LiteLLM client consumes the second. Claude Code, Codex, and fx use the formats in the examples below.

The table is not an exhaustive list of Harbor harnesses. For another harness, use its AI Gateway setup guide and check how the Harbor adapter forwards the connection configuration.

For a custom harness, implement a Harbor agent and pass its Python import path, such as --agent my_package.agent:MyAgent.

Each adapter controls which connection settings it forwards to the harness. Check its endpoint and model handling before switching agents. Changing --agent alone does not translate another adapter's configuration.

The examples run one Terminal-Bench task. Remove --n-tasks 1 to run the full dataset.

Use the Codex compatibility endpoint with Harbor's codex adapter:

harbor run \
  --dataset terminal-bench@2.0 \
  --agent codex \
  --model openai/gpt-6-astra \
  --ae "OPENAI_API_KEY=$AI_GATEWAY_API_KEY" \
  --ae "OPENAI_BASE_URL=https://ai-gateway.vercel.sh/codex/v1" \
  --n-tasks 1

Harbor forwards the API key and writes the base URL into Codex's configuration. Its adapter passes the final model segment, gpt-6-astra, to Codex. AI Gateway resolves that model name. The Codex compatibility endpoint supports the Responses API Codex uses.

Harbor's Claude Code adapter accepts an Anthropic-compatible base URL and preserves the full model name when you configure that URL:

harbor run \
  --dataset terminal-bench@2.0 \
  --agent claude-code \
  --model anthropic/claude-opus-5 \
  --ae "ANTHROPIC_API_KEY=$AI_GATEWAY_API_KEY" \
  --ae "ANTHROPIC_BASE_URL=https://ai-gateway.vercel.sh/claude-code" \
  --n-tasks 1

The adapter passes your AI Gateway key as ANTHROPIC_API_KEY. AI Gateway's Anthropic-compatible API accepts it through the x-api-key header.

Configure OpenCode's OpenAI-compatible provider through its Harbor adapter:

harbor run \
  --dataset terminal-bench@2.0 \
  --agent opencode \
  --model openai/anthropic/claude-opus-5 \
  --ae "OPENAI_API_KEY=$AI_GATEWAY_API_KEY" \
  --ae "OPENAI_BASE_URL=https://ai-gateway.vercel.sh/v1" \
  --n-tasks 1

The first openai/ selects OpenCode's provider. The remaining anthropic/claude-opus-5 is the model ID sent to AI Gateway. Harbor registers the model and sets the provider's baseURL in OpenCode's configuration.

For Pi, specify the API format as well as the base URL:

harbor run \
  --dataset terminal-bench@2.0 \
  --agent pi \
  --model openai/anthropic/claude-opus-5 \
  --ae "OPENAI_API_KEY=$AI_GATEWAY_API_KEY" \
  --ae "OPENAI_BASE_URL=https://ai-gateway.vercel.sh/v1" \
  --ak model_api=openai-completions \
  --n-tasks 1

Harbor writes a custom Pi provider with the endpoint, credential reference, and anthropic/claude-opus-5 model ID. The model_api argument selects the OpenAI Chat Completions format.

Harbor's fx adapter connects directly to AI Gateway and reads AI_GATEWAY_API_KEY from your shell:

harbor run \
  --dataset terminal-bench@2.0 \
  --agent fx \
  --model vercel_ai_gateway/anthropic/claude-opus-5 \
  --n-tasks 1

Harbor installs fx inside the task environment and forwards your credential. The vercel_ai_gateway/ prefix selects the provider; the fx adapter removes it before passing anthropic/claude-opus-5 to fx.

See the fx guide to use fx outside Harbor.

Harbor's fx adapter accepts these credentials from your shell or through --ae KEY=VALUE:

Use an API key for the local example above. OIDC tokens require a valid Vercel identity and have a limited lifetime.

Harbor separates the agent's environment (--ae) from the verifier's environment (--ve). If a task uses a language model to judge results, pass its credentials and endpoint through --ve. Agent credentials do not automatically configure the verifier.

For a verifier that reads OPENAI_API_KEY and OPENAI_BASE_URL, add these flags to harbor run:

--ve "OPENAI_API_KEY=$AI_GATEWAY_API_KEY" \
--ve "OPENAI_BASE_URL=https://ai-gateway.vercel.sh/v1"

Use the variable names and model configuration that the task's verifier expects. A verifier that uses another provider may require different settings.

• Review requests and spend with AI Gateway observability
• Configure spend limits on your API key
• Explore Harbor's source for provider and agent adapter configuration