--- title: How to build your own AI model router description: Build an AI model router with Vercel AI Gateway. Keep routing, key, and retention decisions in your code while the gateway handles provider integrations, failover, and cost metering url: /kb/guide/how-to-build-your-own-ai-model-router canonical_url: "https://vercel.com/kb/guide/how-to-build-your-own-ai-model-router" published: 2026-08-05 last_updated: 2026-08-05 authors: Eric Dodds related: - /docs/ai-gateway - /docs/ai-gateway/models-and-providers/model-fallbacks - /docs/ai-gateway/authentication-and-byok/byok - /docs/ai-gateway/pricing - /docs/ai-gateway/observability-and-spend/custom-reporting - /docs/ai-gateway/observability-and-spend/budgets - /docs/ai-gateway/security-and-compliance/zdr - /docs/ai-gateway/authentication-and-byok install_vercel_plugin: npx plugins add vercel/vercel-plugin --- Model routers front multiple AI providers behind a single API, choosing which model serves each request. Building one from scratch means maintaining provider integrations, a unified request schema, failover, key management, cost metering, and a model catalog, work that one gateway vendor estimates at 6 to 12 months of engineering. [Vercel AI Gateway](https://vercel.com/docs/ai-gateway) handles that infrastructure so you can build only the layer that makes the router yours: your routing policy, your keys, your spend rules, and your retention posture. This guide shows you how to wire each of those decisions to the gateway as configuration. ## Overview In this guide, you'll learn how to: - Point an existing OpenAI, Anthropic, or AI SDK client at AI Gateway - Configure model fallbacks and provider routing per request - Route requests on your own provider credentials (BYOK) - Track cost per request and attribute spend to your users - Cap spend with budgets and enforce zero data retention ## Prerequisites Before you begin, make sure you have: - A [Vercel account](https://vercel.com/signup). - A Node.js project using [AI SDK](https://ai-sdk.dev/), or an SDK from OpenAI or Anthropic > Your router doesn't need to run on Vercel, since the AI Gateway endpoint accepts authenticated requests from any host or runtime. ## How it works Every router responsibility falls into one of two layers. The decision layer covers who gets access, to which models, on whose keys, with what retention posture, and at what budget. Only you can define it. The mechanics layer covers provider integrations, schema unification, failover execution, and cost metering, and it's the same in every router. AI Gateway provides the mechanics as request-level configuration, so each section below names one decision and shows the option that carries it. ## 1\. Point your client at the gateway The gateway exposes an OpenAI-compatible endpoint, so an existing OpenAI SDK client only needs a new base URL: `import OpenAI from 'openai'; const client = new OpenAI({ apiKey: process.env.AI_GATEWAY_API_KEY, baseURL: 'https://ai-gateway.vercel.sh/v1', }); const completion = await client.chat.completions.create({ model: 'anthropic/claude-opus-5', messages: [{ role: 'user', content: 'Write a haiku about TypeScript.' }], });` Anthropic SDK clients work the same way with `https://ai-gateway.vercel.sh` as the base URL. If you use the AI SDK, there is no client to reconfigure. Add a plain model string, and it routes requests through the gateway automatically, and the SDK reads `AI_GATEWAY_API_KEY` from the environment [by default](https://ai-sdk.dev/providers/ai-sdk-providers/ai-gateway#api-key-authentication): `AI_GATEWAY_API_KEY=your_api_key_here` `import { generateText } from 'ai'; export async function GET() { const result = await generateText({ model: 'xai/grok-4.5', prompt: 'Why is the sky blue?', }); return Response.json(result); }` Model strings use the `creator/model-name` format. The creator names who made the model, while the serving provider is chosen separately by routing. The gateway maintains the [model catalog](https://vercel.com/ai-gateway/models) and serves it at `GET /models`, which your router can re-expose as its own. ## 2\. Configure model fallbacks and provider routing Which models to try, in what order, from which providers, is your routing policy. The gateway executes it from a `models` array under `providerOptions.gateway`: `import { streamText } from 'ai'; export async function POST(request: Request) { const { prompt } = await request.json(); const result = streamText({ model: 'anthropic/claude-fable-5', // Primary model prompt, providerOptions: { gateway: { models: ['anthropic/claude-opus-5', 'google/gemini-3.1-pro-preview'], // Fallbacks, tried in order }, }, }); return result.toUIMessageStreamResponse(); }` The response comes from the first model that succeeds. Provider preference works the same way: `providerOptions: { gateway: { order: ['bedrock', 'anthropic'], // Try Bedrock first, then Anthropic only: ['bedrock', 'anthropic'], // Allow only these providers }, },` The `sort` option orders candidates by cost, time to first token, or throughput. If you declare no preference, the gateway picks providers dynamically based on recent uptime and latency, so use the `only` allowlist when there are providers you haven't approved. [Model fallbacks](https://vercel.com/docs/ai-gateway/models-and-providers/model-fallbacks) also retry failed requests across providers automatically underneath your preferences. To see what actually happened, read `modelAttempts` from the response's provider metadata. It records every model and provider tried, with the error, status code, and response time for each attempt, which you can surface in your own debug logs. ## 3\. Route requests on your own provider keys Whose credentials a request rides is a decision you keep down to the individual request. [Bring Your Own Key (BYOK)](https://vercel.com/docs/ai-gateway/authentication-and-byok/byok) carries no markup from AI Gateway and is available on the paid tier. Store credentials with the gateway at the team level, or pass them per request: `import type { GatewayProviderOptions } from '@ai-sdk/gateway'; import { generateText } from 'ai'; const { text } = await generateText({ model: 'anthropic/claude-opus-5', prompt: 'Hello, world!', providerOptions: { gateway: { byok: { anthropic: [{ apiKey: process.env.ANTHROPIC_API_KEY }], }, } satisfies GatewayProviderOptions, }, });` Request-scoped credentials bypass any credentials stored in the dashboard, and you can supply multiple credentials per provider, tried in order. Two defaults matter for your design: - **Availability fallback**: if a request on your credentials fails, the gateway retries it on system credentials, billed against your [AI Gateway credits](https://vercel.com/docs/ai-gateway/pricing?__vercel_draft=1#top-up-your-ai-gateway-credits). This protects the request but overrides your BYOK decision, so account for it in your cost model. - **Budgets don't apply**: BYOK spend is metered separately and never counts toward any budget. To limit spend, enforce limits in your own code. ## 4\. Track cost per request and attribute spend to your users Every response carries a `generationId` in `providerMetadata.gateway` (injected into the first content chunk on streams). Look up the request's cost, tokens, and latency with `getGenerationInfo()`: ``import { gateway, generateText } from 'ai'; const result = await generateText({ model: 'anthropic/claude-opus-5', prompt: 'Explain quantum entanglement briefly', }); const generationId = result.providerMetadata?.gateway?.generationId; const generation = await gateway.getGenerationInfo({ id: generationId }); console.log(`Cost: $${generation.totalCost.toFixed(6)}`);`` On the OpenAI-compatible surface, pass the generation ID to `GET /v1/generation` for the same data. To attribute spend, attach a user ID and tags to each request: `providerOptions: { gateway: { user: 'user-123', tags: ['team:finance', 'feature:summaries'], }, },` Then query spend grouped however your billing needs it. The `/v1/report` endpoint groups by day, user, model, tag, provider, credential type, retention status, or API key name, and the AI SDK wraps it: `import { gateway } from 'ai'; const report = await gateway.getSpendReport({ startDate: '2026-03-01', endDate: '2026-03-25', groupBy: 'model', });` [Custom reporting](https://vercel.com/docs/ai-gateway/observability-and-spend/custom-reporting) is in beta, scoped to your account, and available on Pro and Enterprise plans. Grouping by `credential_type` separates BYOK spend from system-credential spend, which is how you monitor spend. ## 5\. Cap spend with budgets [Budgets](https://vercel.com/docs/ai-gateway/observability-and-spend/budgets) cap AI Gateway spend at three scopes: your whole team, a single project, or an individual API key. Budgets stack, so a request must pass every budget in scope. Attach one to a key at creation: `vercel ai-gateway api-keys create --name my-api-key --budget 10 --refresh-period monthly` Refresh periods are `daily`, `weekly`, `monthly`, or `none` (cumulative). The budget check runs at the start of each request, so the request that crosses the limit still completes, and further requests are rejected with an HTTP `402` until the budget resets or you raise it. Two boundaries shape how you use budgets in a router: - Budgets cap keys, projects, and your team, not your end users. Per-user quotas stay in your code, backed by the `user` attribution and `/v1/report` data above. - BYOK spend is never counted in any budget. Scope a key per customer or surface, watch its spend in the dashboard, and revoke it if its traffic misbehaves. The dashboard shows requests by model, time to first token, token counts, and spend per project and per key, with exportable logs. ## 6\. Set your data retention posture AI Gateway itself does not retain prompts or outputs, per its [zero data retention policy](https://vercel.com/docs/ai-gateway/security-and-compliance/zdr). Provider-side retention is a decision you set team-wide or per request: `providerOptions: { gateway: { zeroDataRetention: true, } satisfies GatewayProviderOptions, },` With the flag set, the request routes only to providers holding zero data retention agreements. If no eligible provider can serve it, the request fails with `no_providers_available` rather than relaxing your posture. ZDR skips your BYOK keys by default, because your provider agreements can differ from the ZDR agreements Vercel negotiated for system credentials. If you hold your own ZDR agreement with a provider, mark that key as ZDR-compliant in the dashboard to include it in the ZDR routing set. ## 7\. Build the product layer What remains is ordinary product engineering: authentication in front, account and rate rules in the middle, packaging on top. Every gateway request requires [authentication](https://vercel.com/docs/ai-gateway/authentication-and-byok) by API key or OIDC token. If you deploy your router on Vercel, the OIDC token is available automatically as `VERCEL_OIDC_TOKEN`, so the router stores no gateway secrets. Note that an API key in the environment takes precedence over the OIDC token, even when the key is invalid, so remove stale keys from your environment. ## Best practices - **Set** `**only**` **when provider approval matters**. Default routing can select any provider serving the model. The allowlist is the control when some providers aren't approved for your traffic. - **Log** `**modelAttempts**` **on failures**. It's the raw material for your own observability and for surfacing routing decisions to users. - **Monitor BYOK spend through reporting, not budgets**. Group `/v1/report` by `credential_type` and enforce your own limits in code. - **Mark BYOK keys ZDR-compliant deliberately**. Only do this when you hold a zero data retention agreement with that provider. ## Next steps - Read the [AI Gateway documentation](https://vercel.com/docs/ai-gateway) for the full API surface - Configure [model fallbacks](https://vercel.com/docs/ai-gateway/models-and-providers/model-fallbacks) and provider options - Set up [BYOK credentials](https://vercel.com/docs/ai-gateway/authentication-and-byok/byok) for your team - Explore [budgets](https://vercel.com/docs/ai-gateway/observability-and-spend/budgets) and [custom reporting](https://vercel.com/docs/ai-gateway/observability-and-spend/custom-reporting) for spend control