Teams add model providers one at a time. An app ships on one model, then a second and third get added for different tasks. Each addition brings another API key, another retention policy, and billing fragments across providers.
A secure AI gateway reverses that drift. This guide covers what a secure AI gateway controls, how a request flows through one, and why it helps to have the gateway at the first deployment rather than the first outage.
Key takeaways:
A secure AI gateway centralizes the decisions that direct API calls scatter across services, including which provider sees each prompt and where a request goes after failure. It also moves retention policy into the same control plane.
Vercel's secure AI Gateway layer deletes prompts and outputs immediately after inference. Enforcing zero data retention (ZDR) at the provider level is an opt-in control that routes only to providers with negotiated agreements.
Apps deployed on Vercel authenticate to AI Gateway with an automatic OpenID Connect (OIDC) token, so no provider API keys live in application code.
Copy link to headingWhat is a secure AI gateway?
A secure AI gateway is a routing layer that sits between application code and model providers. It normalizes requests and centralizes the operational controls that direct API calls leave scattered across services. It governs which provider sees a given request, where that request goes when a provider fails, whether the provider keeps the data after inference completes, and what the call costs.
Copy link to headingWhat’s the difference between a secure AI gateway and a standard API gateway?
A standard API gateway manages traffic between clients and backend services. A secure AI gateway manages traffic between applications and large language model (LLM) providers through model-aware routing and token-based rate limiting.
It decides, per request, who is allowed to see the prompt and enforces that decision before the request leaves your network.
The practical differences show up on four axes worth checking before you pick one:
The AI gateway earns the "secure" qualifier on that last row with provider selection, retention enforcement, and per-request audits.
Copy link to headingWhat data routing controls a secure AI gateway governs
The data routing controls are what a secure AI gateway exists to enforce. They govern data exposure through four concrete abilities:
Provider selection: Decide which provider sees a given request, so sensitive workloads never touch a provider you have not reviewed.
Path blocking: Deny requests for a named model outright, which stops a team from calling a model that has not been approved.
Retention enforcement: Require zero data retention at the provider level, so a compliant provider deletes the data after inference.
Routing audit: Record which providers were considered and filtered for each request so that a routing decision can be reconstructed after the fact.
Direct provider calls leave those controls outside the application, which produces a consistent set of failure modes:
Credential sprawl: Each provider's API key or token is embedded in application code, one leak away from exposure.
Provider lock-in: Swapping a model means a code change and a redeploy, because provider-specific paths accumulate in every service.
No failover: Applications that call providers directly absorb outages and rate limits as errors instead of routing decisions.
Spend invisibility: Token usage and cost scatter across application logs, so nobody can say which service is burning through budget.
Uncontrolled prompt data: No built-in mechanism for access control, audit trails, or retention enforcement exists once the request leaves your network.
Each failure mode traces back to the same root cause: a decision made per service instead of once. The next section follows a single request through a gateway to show where those decisions get made instead.
Copy link to headingHow an AI gateway routes, filters, and audits model requests
Consolidating those decisions only helps if the routing path itself is fast and observable.
Copy link to headingHow a request travels from edge to provider
A request through Vercel's AI Gateway makes four hops. It lands on one of 126 points of presence (PoPs) worldwide, travels Vercel's private backbone to the nearest of 20 compute regions, and reaches the gateway, which runs as a Vercel Function on Fluid compute.
From there, the function authenticates the caller, verifies quotas through in-region Redis, and prepares the payload before the final hop to the provider. Every hop stays inside Vercel-managed boundaries, which avoids unpredictable public internet routes.
Copy link to headingHow the gateway authenticates callers without provider keys
Authentication is where credential sprawl gets solved. For apps deployed on Vercel, an OIDC token is available automatically as VERCEL_OIDC_TOKEN, so there are no secrets to manage and no keys to rotate. Application code authenticates to the gateway, and the gateway authenticates to providers, so provider credentials exist in one place.
Copy link to headingWhich routing and retention controls each plan includes
The controls themselves divide by mechanism, plan, and cost.
The plan column shows which controls are available before an upgrade:
Time to first token (TTFT) and tokens per second (TPS) are the two performance dimensions the provider sort ranks on, alongside cost.
Copy link to headingHow zero data retention works at the gateway and provider layers
Zero data retention (ZDR) deserves a careful reading, because it operates at two layers. The gateway layer does not retain prompts, outputs, or sensitive data, and user data is deleted immediately after a request completes. Provider-level retention is separate, and a provider's default policy may differ from the agreement Vercel has in place.
Enforcing ZDR at the provider level is opt-in. When active, the gateway routes only to providers with negotiated ZDR agreements. Any provider with unknown status is treated as non-compliant and skipped. Each response includes gateway.routing.planningReasoning metadata, an audit trail of which providers were considered and filtered.
When team-wide ZDR is on, bring your own key (BYOK) credentials are skipped by default, because your provider agreements can differ from Vercel's with no visibility into them. You can mark a BYOK key as ZDR-compliant manually, and responsibility for verifying the underlying contract then transfers to you. For regulated workloads, skipping is the correct default.
Copy link to headingRouting a model call through the gateway
Routing rule changes propagate without code changes, and in-flight requests finish under the previous configuration.
On AI SDK, a plain model string routes through AI Gateway by default with no provider wiring.
That default is what turns the controls above from configuration into behavior every request inherits.
Copy link to headingWhy data routing controls are important from day one
Teams treat AI gateways as an enterprise concern, something added once traffic spans multiple providers or a dedicated infrastructure team exists. The evidence points the other way.
Three exposures start on day one:
Provider uptime: Anthropic's documented 90-day uptime runs in the 99.1 to 99.4% range across services, roughly 19 to 23 hours of degraded service per quarter per service. In August and September 2025, three infrastructure bugs degraded Claude response quality over weeks, each producing different symptoms on different platforms. A team calling a provider directly absorbs that as production errors.
Data protection: The European Data Protection Board (EDPB) classifies the deployer of an LLM as the data controller and the gateway as a processor. Provider default retention sits outside your control on a direct call.
Routing controls are the baseline. A direct API call is the pattern that needs justification, because it accepts every one of these exposures by default the moment the first request ships.
Copy link to headingHow Vercel AI Gateway enforces data routing controls
Each of those day-one exposures maps to a control Vercel AI Gateway enforces, and the production data shows what changes once it does. Credential sprawl is where it starts.
Copy link to headingUnified provider access without credential sprawl
The first team to add a second model provider learns that provider-specific code paths multiply fast. Each key lives in a different service, swapping a model means a redeploy, and no single place holds the answer to which providers the app can reach. That sprawl is a security surface before it is an inconvenience.
Vercel collapses it to one control plane. Application code holds a single OIDC token, the gateway holds the provider credentials, and a model swap happens through a routing rule rather than a code change.
This is the same system that powered v0.app for millions of users before external general availability and now processes billions of tokens per day. Hence, the consolidation runs at production scale, not in a demo.
Copy link to headingAutomatic failover when a provider goes down
Applications that call a provider directly have no answer when that provider degrades. The outage surfaces to the user as an error, and the on-call engineer discovers it from a dashboard rather than routing around it. Given provider uptime in the 99.1 to 99.4% range, that failure is not an edge case but a recurring quarterly event.
AI Gateway reroutes instead. Across gateway traffic, the rescue rate runs at 5.1% measured in tokens and 4.9% in dollars.
Roughly 1 in 20 token-dollars would have surfaced as an error without gateway-level failover. Cline, an open-source coding agent, ran a week of live A/B testing that split production traffic between its previous router and AI Gateway. P99 streaming latency improved 10 to 14% across its most-used models, and API error rates dropped 43.8%.
Copy link to headingZero data retention and provider allowlists
Once a request leaves your network on a direct call, retention is whatever the provider's default policy says, and a regulated workload has no mechanism to prove otherwise. The compliance question then blocks the launch, because nobody can attest to where the prompt went.
AI Gateway answers it at two layers. The gateway layer deletes prompts and outputs immediately after inference. Provider-level ZDR, enabled per request or team-wide, routes only to providers with negotiated agreements while skipping any provider of unknown status.
A provider allowlist restricts routing to the set your team has reviewed, and the planningReasoning metadata records which providers were filtered. The tradeoff is that team-wide ZDR skips BYOK keys by default and costs $0.10 per 1,000 successful requests, so you trade a per-request fee and manual key verification for an attestable retention posture.
Copy link to headingSpend controls and routing observability
Token spend is invisible at the call level until the invoice arrives. By then, the runaway loop or the context-window growth has already run for a billing cycle. Direct integrations scatter the cost data across service logs, so the answer to which service burned the budget takes an investigation.
AI Gateway consolidates spend and routing into one view. Per-key budgets return HTTP 402 on breach before the spend reaches the invoice, and the planningReasoning audit trail explains every routing decision.
Searchable has processed 100 billion tokens through the gateway with zero model SDK implementation and zero API key rotation. They report a 5x increase in development velocity with customer-requested features shipped in as little as 30 minutes. Removing the per-provider wiring returns time to the team.
Copy link to headingMake secure data routing your default on Vercel
The direct-API-call model distributes credential management, retention policy, failover logic, and spend controls across every service that calls a model. Each wiring decision stays invisible to the others. That complexity is the real cost, and it compounds with every provider added. Centralizing those decisions into one auditable control plane removes it.
Here is how Vercel AI Gateway makes secure data routing the default rather than a project:
One OIDC token instead of provider keys: A single automatic token replaces a drawer of provider API keys, so no credential lives in application code.
Zero data retention at the routing layer: The gateway deletes prompts and outputs after inference, and provider-level ZDR routes only to providers with negotiated agreements.
Routing rules without a deploy: Rewrite and deny rules change which model serves a request and propagate without a code change, while in-flight requests finish under the previous configuration.
Automatic failover across providers: Gateway-level rerouting turns provider outages into routing decisions instead of user-facing errors, with a measured rescue rate of 5.1% of tokens.
Per-key budgets that cap spend: Budget caps return HTTP 402 on breach, so runaway usage stops before it reaches the invoice.
Start a new project at vercel.com/new to route your first model call through AI Gateway, or browse vercel.com/templates for AI apps already wired up for production.
Copy link to headingFAQs about secure AI gateways
Copy link to headingDoes a secure AI gateway store my prompts and outputs?
Vercel's gateway layer does not. It has a zero-data-retention policy that permanently deletes prompts, outputs, and sensitive data after a request completes. Provider-level retention is separate, and ZDR is applied per-request or team-wide only to providers with negotiated agreements.
Copy link to headingWhat happens if I turn on ZDR and also use BYOK?
BYOK keys are skipped by default when ZDR is active, because Vercel cannot verify that your provider agreement covers zero data retention. You can manually mark a BYOK key as ZDR-compliant, and you then assume responsibility for confirming the underlying contract before doing so.
Copy link to headingWhat is a routing rule, and how does it differ from a provider sort?
A routing rule rewrites a request from one model to another or denies a model with an HTTP 403, applied to every request made with your team's credentials. A provider sort orders the providers behind a model by cost, TTFT, or TPS without blocking any of them.
Copy link to headingHow does AI Gateway handle SOC 2 and GDPR compliance?
Vercel holds SOC 2 Type 2 attestation covering Security, Confidentiality, and Availability, along with ISO 27001, PCI DSS v4.0, and a General Data Protection Regulation (GDPR) compliant data processing agreement (DPA). HIPAA business associate agreements (BAAs) are available on Enterprise. No dedicated AI Gateway compliance scope statement has been published, so teams with regulated workloads should confirm scope with Vercel directly.