How to build your own AI model router

Model routers front multiple AI providers behind a single API, choosing which model serves each request.

If AI is integrated into your product, offering users a choice in the models they use can help them optimize for better output, lower costs, or more intelligent workflows. Users can also experiment with the latest models when they are released, and routing can be offered as a stand-alone product itself.

Building a model router from scratch means maintaining provider integrations, a unified request schema, failover, key management, cost metering, and a model catalog, work that one gateway vendor estimates at 6 to 12 months of engineering.

Vercel AI Gateway handles that infrastructure so you can build only the layer that makes the router yours: your routing policy, your keys, your spend rules, and your retention posture. This guide shows you how to wire each of those decisions to the gateway as configuration.

Copy link to headingOverview

In this guide, you'll learn how to:

• Point an existing OpenAI, Anthropic, or AI SDK client at AI Gateway
• Configure model fallbacks and provider routing per request
• Route requests on your own provider credentials (BYOK)
• Track cost per request and attribute spend to your users
• Cap spend with budgets and enforce zero data retention

Copy link to headingPrerequisites

Before you begin, make sure you have:

• A Vercel account
• A Node.js project using AI SDK, or an SDK from OpenAI or Anthropic

Copy link to headingHow it works

Every router responsibility falls into one of two layers:

1. The decision layer covers who gets access, to which models, on whose keys, with what retention posture, and at what budget. This layer defines the end-user experience in your product.
2. The mechanics layer covers provider integrations, schema unification, failover execution, observability, and cost metering.

AI Gateway provides the mechanics as request-level configuration. The guide below walks through setting up your model router on AI Gateway.

Copy link to heading1. Point your client at the gateway

The gateway exposes an OpenAI-compatible endpoint, so an existing OpenAI SDK client only needs a new base URL:

import OpenAI from 'openai';

const client = new OpenAI({
  apiKey: process.env.AI_GATEWAY_API_KEY,
  baseURL: 'https://ai-gateway.vercel.sh/v1',
});

const completion = await client.chat.completions.create({
  model: 'anthropic/claude-opus-5',
  messages: [{ role: 'user', content: 'Write a haiku about TypeScript.' }],
});

Anthropic SDK clients work the same way with https://ai-gateway.vercel.sh as the base URL.

If you use the AI SDK, there is no client to reconfigure. Add a plain model string to route requests through the gateway automatically. The SDK reads AI_GATEWAY_API_KEY from the environment by default:

AI_GATEWAY_API_KEY=your_api_key_here

import { generateText } from 'ai';

export async function GET() {
  const result = await generateText({
    model: 'xai/grok-4.5',
    prompt: 'Why is the sky blue?',
  });
  return Response.json(result);
}

Model strings use the creator/model-name format.

The creator names who made the model, while the serving provider is chosen separately by routing. The gateway maintain an up-to-date model catalog and serves it at GET /models, which your router can re-expose as its own.

Copy link to heading2. Configure model fallbacks and provider routing

A routing policy defines which models to use, in what order, from which providers. The gateway executes it from a models array under providerOptions.gateway:

import { streamText } from 'ai';

export async function POST(request: Request) {
  const { prompt } = await request.json();

  const result = streamText({
    model: 'anthropic/claude-fable-5', // Primary model
    prompt,
    providerOptions: {
      gateway: {
        models: ['anthropic/claude-opus-5', 'google/gemini-3.1-pro-preview'],
      },
    },
  });

  return result.toUIMessageStreamResponse();
}

The response comes from the first model that succeeds.

Provider preference works the same way:

providerOptions: {
  gateway: {
    order: ['bedrock', 'anthropic'], // Try Bedrock first, then Anthropic
    only: ['bedrock', 'anthropic'],  // Allow only these providers
  },
},

The sort option orders candidates by cost, time to first token, or throughput.

If you declare no preference, the gateway picks providers dynamically based on recent uptime and latency, so use the only allowlist when there are providers you haven't approved. Model fallbacks also retry failed requests across providers automatically underneath your preferences.

To see what actually happened when a model was requested, read modelAttempts from the response's provider metadata. It records every model and provider tried, with the error, status code, and response time for each attempt, which you can surface in your own debug logs.

Copy link to heading3. Route requests on your own provider keys

Whose credentials a request rides is a decision you can apply at the individual request level. AI Gateway’s Bring Your Own Key (BYOK) feature carries no markup from AI Gateway and is available on the paid tier.

Store credentials with the gateway at the team level, or pass them per request:

import type { GatewayProviderOptions } from '@ai-sdk/gateway';
import { generateText } from 'ai';

const { text } = await generateText({
  model: 'anthropic/claude-opus-5',
  prompt: 'Hello, world!',
  providerOptions: {
    gateway: {
      byok: {
        anthropic: [{ apiKey: process.env.ANTHROPIC_API_KEY }],
      },
    } satisfies GatewayProviderOptions,
  },
});

Request-scoped credentials bypass any credentials stored in the dashboard, and you can supply multiple credentials per provider, tried in order.

Two defaults matter for your design:

• Availability fallback: if a request on your credentials fails, the gateway retries it on system credentials, billed against your AI Gateway credits. This protects the request but overrides your BYOK decision, so account for it in your cost model.
• Budgets don't apply: BYOK spend is metered separately and never counts toward any budget. To limit spend, enforce limits in your own code.

Copy link to heading4. Track cost per request and attribute spend to your users

Every response carries a generationId in providerMetadata.gateway (injected into the first content chunk on streams).

Look up the request's cost, tokens, and latency with getGenerationInfo():

import { gateway, generateText } from 'ai';

const result = await generateText({
  model: 'anthropic/claude-opus-5',
  prompt: 'Explain quantum entanglement briefly',
});

const generationId = result.providerMetadata?.gateway?.generationId;
const generation = await gateway.getGenerationInfo({ id: generationId });

console.log(`Cost: $${generation.totalCost.toFixed(6)}`);

On the OpenAI-compatible surface, pass the generation ID to GET /v1/generation for the same data.

To attribute spend, attach a user ID and tags to each request:

providerOptions: {
  gateway: {
    user: 'user-123',
    tags: ['team:finance', 'feature:summaries'],
  },
},

Then query spend grouped however your billing needs it.

The /v1/report endpoint groups by day, user, model, tag, provider, credential type, retention status, or API key name, and the AI SDK wraps it:

import { gateway } from 'ai';

const report = await gateway.getSpendReport({
  startDate: '2026-03-01',
  endDate: '2026-03-25',
  groupBy: 'model',
});

Custom reporting is in beta, scoped to your account, and available on Pro and Enterprise plans. Grouping by credential_type separates BYOK spend from system-credential spend, which is how you monitor spend.

Copy link to heading5. Cap spend with budgets

Budgets cap AI Gateway spend at three scopes: your whole team, a single project, or an individual API key. Budgets stack, so a request must pass every budget in scope. Attach one to a key at creation:

vercel ai-gateway api-keys create --name my-api-key --budget 10 --refresh-period monthly

Refresh periods are daily, weekly, monthly, or none (cumulative).

The budget check runs at the start of each request, so the request that crosses the limit still completes, and further requests are rejected with an HTTP 402 until the budget resets or you raise it.

Two boundaries shape how you use budgets in a router:

• Budgets cap keys, projects, and your team, not your end users. Per-user quotas stay in your code, backed by the user attribution and /v1/report data above.
• BYOK spend is never counted in any budget.

Scope a key per customer or surface, watch its spend in the dashboard, and revoke it if its traffic misbehaves. The dashboard shows requests by model, time to first token, token counts, and spend per project and per key, with exportable logs.

Copy link to heading6. Set your data retention posture

AI Gateway itself does not retain prompts or outputs, per its zero data retention policy. Provider-side retention is a decision you set team-wide or per request:

providerOptions: {
  gateway: {
    zeroDataRetention: true,
  } satisfies GatewayProviderOptions,
},

With the flag set, the request routes only to providers holding zero data retention agreements. If no eligible provider can serve it, the request fails with no_providers_available rather than relaxing your posture.

ZDR skips your BYOK keys by default, because your provider agreements can differ from the ZDR agreements Vercel negotiated for system credentials. If you hold your own ZDR agreement with a provider, mark that key as ZDR-compliant in the dashboard to include it in the ZDR routing set.

Copy link to heading7. Build the product layer

What remains is ordinary product engineering: authentication in front, account and rate rules in the middle, packaging on top.

Every gateway request requires authentication by API key or OIDC token. If you deploy your router on Vercel, the OIDC token is available automatically as VERCEL_OIDC_TOKEN, so the router stores no gateway secrets. Note that an API key in the environment takes precedence over the OIDC token, even when the key is invalid, so remove stale keys from your environment.

Copy link to headingBest practices

• Set only when provider approval matters. Default routing can select any provider serving the model. The allowlist is the control when some providers aren't approved for your traffic.
• Log modelAttempts on failures. It's the raw material for your own observability and for surfacing routing decisions to users.
• Monitor BYOK spend through reporting, not budgets. Group /v1/report by credential_type and enforce your own limits in code.
• Mark BYOK keys ZDR-compliant deliberately. Only do this when you hold a zero data retention agreement with that provider.

Copy link to headingNext steps

• Read the AI Gateway documentation for the full API surface
• Configure model fallbacks and provider options
• Set up BYOK credentials for your team
• Explore budgets and custom reporting for spend control