The same sequence plays out across teams shipping large language model (LLM) features. A team integrates one provider, adds a second for cost reasons, then gets paged when the first provider goes down.
Provider selection has become an infrastructure problem, and the gateway layer is where this selection gets resolved or compounded. The outcome depends on which reference patterns a team implements and in what order.
This guide covers the reference patterns important for a production gateway, which three to build before anything else, and how those choices hold up once traffic grows.
Key takeaways:
Provider outages are routine, and a gateway absorbs them before application code ever sees an error.
A production gateway needs unified provider abstraction and automatic failover before per-request cost attribution, and the remaining patterns wait for specific scale or compliance requirements.
Semantic caching adds an embedding call to every request, while production routing data points to failover and multi-model routing as the more durable operational investment.
Cost attribution has to live in the routing layer, because the gateway is the one place every request already passes through.
Copy link to headingWhat is AI gateway architecture?
AI gateway architecture is the design of the routing layer that sits between an application and the LLM providers it calls. It exposes one interface to application code and handles provider translation, authentication, failover, and usage accounting behind that interface. Provider selection becomes a configuration decision instead of a code change.
The layer captures more than which provider served a request. It owns the fallback behavior when a provider degrades, the credential handling that keeps keys out of application code, and the per-request accounting that makes token spend attributable. Those responsibilities are why a gateway is infrastructure rather than a helper library.
Copy link to headingWhy provider fragmentation forces a gateway layer
Provider fragmentation forces a gateway layer because every provider integrated by hand adds duplicate code that application logic should never own. Each LLM provider ships its own software development kit (SDK), authentication scheme, request format, error codes, and rate-limit behavior.
Add a second provider without a gateway and that provider-specific logic spreads through the codebase, which is the point where provider selection stops being a product decision and becomes an infrastructure one.
Reliability makes the case sharper. Provider outages are routine and Vercel's AI Gateway reports uptime as a rolling endpoint-level metric over windows such as the last 15 minutes, 1 hour, and 1 day. A gateway that reroutes on those signals absorbs the outage before application code returns an error. The patterns below are how that gets built.
Copy link to headingThe three patterns every AI gateway needs first
Ten reference patterns get discussed for gateways, but three carry a deployment to production readiness. Start with unified provider abstraction and automatic failover, then add per-request cost attribution.
The remaining seven patterns, including semantic caching, traffic mirroring, two-tier architecture, and multi-region shared state, answer specific scale, compliance, or evaluation requirements. Teat them as later additions rather than launch blockers.
Copy link to headingUnified provider abstraction
Unified provider abstraction is a single gateway endpoint that translates one application-facing interface into each provider's own format, so application code targets one interface no matter which model serves the request.
The gateway exposes a single endpoint and does provider-specific translation internally, so application code targets one interface no matter which model serves the request. The tradeoff is a network hop plus the ongoing burden of tracking provider application programming interface (API) changes as each vendor revises formats and error codes. In exchange, adding or swapping a provider collapses to a configuration change instead of a code path.
Copy link to headingAutomatic failover
Automatic failover is an ordered fallback chain that reroutes a request to the next model or provider when the primary one fails, rate-limits, or times out, without the application seeing an error.
The gateway holds an ordered fallback chain and reroutes transparently when a primary model or provider fails, rate-limits, or times out. On Vercel's platform, roughly 3.5% of requests complete after a fallback. The rescue rate runs higher when weighted by cost at 5.1% of tokens and 4.9% of dollars. Those are requests that would have failed or degraded without the gateway in front of them.
Copy link to headingPer-request cost attribution
Per-request cost attribution is the practice of recording the model, tokens, and dollar cost of every individual request at the gateway, so token spend is traceable to a user, feature, or key instead of surfacing as one lump invoice.
Without per-request cost attribution, token spend stays invisible until the invoice arrives. The failure has four common roots: opaque per-model token pricing, shared keys with no attribution, runaway agent loops, and no per-request audit trail. The gateway is the natural place to fix all four, because every request already passes through it.
Provider abstraction and failover make the gateway reliable, and cost attribution makes it accountable. The other seven patterns earn their place later, once a specific requirement calls for them.
Copy link to headingWhy production gateways should prioritize failover over semantic caching
Semantic caching is the pattern teams reach for first and the one production data supports least. Semantic caching embeds every incoming prompt as a vector, queries a vector store for similar prior requests, and returns a cached response only when similarity clears a threshold.
That is an embedding-model call on 100% of requests to skip the LLM call on some fraction of them.
Set the threshold loose and the gateway returns contextually wrong responses. Set it tight and the hit rate drops toward zero.
Failover and multi-model routing compound in the other direction. Vercel’s 3.5% fallback-completion rate covers real requests kept alive at production token volumes, and that reliability accrues on every request, not only when a cache happens to hit.
The cost surfaces when provider errors, readiness probes, database writes, and fallback accounting all land on the on-call path. Teams end up testing for cascading failures under load and throughput regressions with the proxy in the path.
Self-hosting makes sense at high enough spend with the engineering time to operate it, and a managed gateway makes sense when that time is worth more elsewhere.
Copy link to headingHow AI gateway architecture scales with request volume
Routing reliability matters more as traffic grows, and so does the number of models a team runs. Vercel's production index shows model count tracking request volume across the platform.
The table below shows the average number of distinct models a team routes across at each volume tier:
Nobody plans for 35 models. Teams route to three, then five, then eight, and the complexity accumulates alongside usage and the complexity accumulates with usage, never as a deliberate choice.
Copy link to headingHow Vercel's AI Gateway implements these patterns
Vercel AI Gateway applies these patterns on the same infrastructure that runs the application calling it. It was the routing layer behind v0 before it became a standalone product, so production load shaped its design. It now routes enough daily traffic to fill a monthly production index of aggregate patterns.
Copy link to headingProvider abstraction through one model string
Adding a second provider by hand means duplicating auth, request formatting, and error handling for each one, which is the fragmentation problem in miniature. AI Gateway keeps that logic out of application code. In AI SDK, a model string in creator/model-name format routes through the gateway with no additional configuration.
Passing a model string to streamText routes the call through the gateway:
import { streamText } from 'ai';
const result = streamText({ model: 'openai/gpt-5.5', prompt: 'Summarize the incident report.',});
Switching providers means changing that string, not the surrounding code.
Copy link to headingFailover without application code changes
Getting paged because a primary provider degraded is the failure the fragmentation problem produces at 3 a.m. AI Gateway answers it with fallback chains configured through a models array in providerOptions.gateway, tried in order until one succeeds or the options run out. From the application's perspective, the request succeeded.
Adding fallback models keeps the request alive when the primary fails:
const result = streamText({ model: 'openai/gpt-5.5', prompt: 'Summarize the incident report.', providerOptions: { gateway: { models: ['anthropic/claude-opus-4.6', 'google/gemini-3.1-pro-preview'], }, },});Zo Computer ran on custom adapter code at a 99.3% success rate and a 20x reduction in retry rate, which added up to tens of thousands of model fallbacks a day. Moving to AI SDK and AI Gateway dropped the retry rate to 0.34% and cut P99 latency by 38%.
Copy link to headingRouting on the same infrastructure as the application
Routing that leaves managed infrastructure adds latency a team cannot see or control. AI Gateway runs AI Gateway is a Node.js service on Fluid compute and Vercel's ontent delivery network (CDN), which spans 126 Points of Presence and 20 compute regions. Requests travel across a private backbone instead of the public internet, and through secure persistent tunnels.
The workload shape also drove the pricing model. In its first month, AI Gateway handled roughly 16,000 runtime hours, but only 1,200 of those involved actual CPU work, with the rest spent waiting on providers.
That ratio, under 8% of runtime, is why Fluid Compute bills Active CPU rather than wall-clock time. Cline measured the routing gain directly, running a week of live A/B testing against its previous router and recording 13.7% faster P99 latency on Grok Code Fast 1 and 14.4% faster on Minimax M2.
Copy link to headingCost attribution and spend controls in the routing layer
Shared keys and opaque pricing are why spend goes untracked until the bill lands. Because every request passes through the gateway, attribution belongs there. The Custom Reporting API breaks down cost and token consumption by model, user, tag, provider, or credential type, returning fields including total_cost, market_cost, input_tokens, output_tokens, cached_input_tokens, reasoning_tokens, and request_count.
One platform serving more than 200,000 users consolidated its third-party proxy tracking into AI Gateway and saved over $80,000.
Enforcement sits next to attribution. Setting a spend cap on a key makes AI Gateway reject further requests on that key until the budget resets.
Creating a budgeted key through the Vercel command-line interface (CLI) takes one command:
vercel ai-gateway api-keys create --name checkout-agent --budget 500 --refresh-period monthly --include-byokApplications on Vercel authenticate to the gateway with OpenID Connect (OIDC) tokens, and the gateway injects provider credentials at routing time, so application code never sees the underlying keys.
Bring your own key (BYOK) credentials are scoped at the team level with no markup on inference. One detail belongs in any BYOK cost model: when a BYOK credential fails, AI Gateway retries with system credentials and bills that usage against the team's credits balance, and that fallback cannot currently be disabled.
Copy link to headingShip production LLM features with Vercel
The fragmentation problem starts small, with one provider and a second added for cost, and it turns into pages, invoice surprises, and provider logic scattered across the codebase. The reference patterns move that problem into the routing layer, and the order matters: provider abstraction and failover first, then cost attribution, then the rest as scale demands.
Vercel AI Gateway collapses those patterns onto managed infrastructure teams can adopt without building their own routing layer, offering:
Unified provider access: One model string in
creator/model-nameformat routes across more than 200 models and 35-plus providers through the AI SDK, with no per-provider code.Automatic model fallbacks: An ordered
modelsarray inproviderOptions.gatewayreroutes on failure transparently, which rescues roughly 3.5% of requests that would otherwise fail.Custom Reporting and budgets: Per-request cost attribution by model, user, tag, and provider, with per-key spend caps that reject requests once a budget is exceeded.
Fluid compute with Active CPU pricing: Sub-20-millisecond routing on a private backbone, billed for active CPU rather than the time spent waiting on providers.
OIDC and BYOK: Provider credentials injected at routing time and scoped to the team, keeping keys out of application code with no markup on inference.
Start a new project to route your first model call through AI Gateway at vercel.com/new, or browse vercel.com/templates for AI applications already wired up for production.
Copy link to headingFrequently asked questions about AI gateway architecture
Copy link to headingDoes AI gateway architecture work with providers beyond OpenAI and Anthropic?
Yes. A gateway abstracts the provider behind one interface, so it can route to any supported vendor. Vercel AI Gateway supports more than 200 models across 35-plus providers, including Google, xAI, Groq, and MiniMax, using a creator/model-name model string.
Copy link to headingWhen a BYOK request fails, does the fallback use my credentials or Vercel's?
Vercel's. AI Gateway retries the failed request with system credentials and bills that fallback usage against your team's credits balance. The behavior cannot currently be turned off, so factor it into any BYOK-only cost model rather than assuming failures stay on your own keys.
Copy link to headingHow does an AI gateway handle failover without application code changes?
The gateway holds an ordered fallback chain and reroutes on failure internally. In AI SDK, fallbacks live in a models array in providerOptions.gateway, tried in order until one succeeds. The application sees a successful response and never runs provider-specific retry logic.
Copy link to headingIs there an overhead cost to routing through an AI gateway?
There is a network hop, but it is small relative to LLM latency. Vercel AI Gateway keeps routing overhead under 20 milliseconds, while model inference and output generation take far longer. The overhead matters most for high-frequency, low-latency work like real-time embeddings.
Copy link to headingWhen does self-hosting an AI gateway make more sense than a managed one?
Self-hosting fits teams with real DevOps capacity, predictable traffic, and a clear owner for provider changes, and it pays off mainly at high monthly LLM spend. Below that, operating a proxy such as LiteLLM can cost more in engineering time than a managed gateway.