A large language model (LLM) proxy is a server-side intermediary between a web app and model providers. At minimum, it is one server route that holds the provider key and forwards model calls, so the key never reaches the browser. A proxy can handle basic forwarding when routing and access requirements are limited.
Adding a second provider, a shared key, or a monthly budget can expand the proxy's responsibilities. Requests may now need provider selection, retries when a provider fails, and costs tracked against someone's limit. A minimal forwarding proxy does not include that logic, so it can spread across route handlers.
Vercel AI Gateway provides unified model access through a managed API, with provider routing, usage tracking, and billing in one place. You can use it from applications hosted anywhere, and the AI SDK can authenticate automatically with short-lived tokens on Vercel. This guide covers what a minimal forwarding proxy handles, what a gateway can add, and when switching is worthwhile.
Copy link to headingWhat is an LLM proxy?
An LLM proxy is a server-side reverse proxy for model APIs. Your app sends model requests to a single endpoint, and the proxy forwards them to the provider hosting the requested model. Provider keys live in the proxy's environment or secret store, so the browser and client code never see them.
A minimal proxy only forwards requests. Retries, spend caps, and request logs are additional controls that require configuration, maintenance, and sometimes storage.
Copy link to headingHow is an LLM proxy different from an AI gateway?
A minimal proxy forwards each request to a provider and returns the provider's response. An AI gateway can also choose which provider gets the request, how much it is allowed to cost, and what gets logged.
Tools don't use these names consistently, so it's more reliable to look at what each one does with a request. The table compares a minimal forwarding proxy with gateway capabilities that vary by product and configuration:
Model providers return errors, rate limits, and timeouts in production. A gateway can handle those failures. Vercel AI Gateway’s April 2026 production index reported that about 3.5% of requests hit one of those provider problems on the first try and still succeeded because the gateway rerouted them. Without a gateway, SDK, or application retries, those initial failures would have reached the app.
Copy link to headingWhy does an LLM proxy matter for web apps?
Browser code can't safely hold a long-lived provider key, so web apps that use those keys need a server-side hop. A minimal proxy provides this hop and offers benefits that a direct browser call with a provider key cannot:
Key secrecy: The provider key lives in a server environment variable or secret store, so a page visitor can't copy it and run requests billed to your account.
Revocable client credentials: Proxies with client-key management let backend callers authenticate with their own keys, so cutting off one teammate or script doesn't mean rotating the provider key. Browser users can authenticate through the app's session instead.
Drop-in client compatibility: A proxy that speaks the OpenAI request format lets compatible clients switch by changing the base URL, key, and model name.
One place for controls: Rate limits, abuse checks, and usage tracking can run on the same server.
Copy link to headingCore components of an LLM proxy server
An LLM proxy server consists of an endpoint, a forwarder, and a key store. Each one can start small and gets more expensive to run as more people depend on it.
Copy link to headingOne endpoint for every model call
Your app sends model calls to a single URL instead of talking to each provider's API directly. OpenAI-compatible proxy endpoints use paths like /chat/completions and /embeddings, usually under a /v1 base path. Proxies with client-key management can authenticate your app with a proxy-issued key instead of the provider key.
With an OpenAI-compatible proxy, the OpenAI format is your API contract. Other providers' requests and responses may need translation, and your code depends on that translation layer. A separate internal proxy can be restricted to your backend. If browsers call an application API, authenticate users and enforce authorization, rate limits, and input validation on the server.
Copy link to headingA forwarder that maps models to providers
The forwarder takes the model name your app asks for, sends the request to the provider behind it, and passes the response back. If the proxy supports model aliases, you can point an alias to a different provider in the config without touching the app code.
Nearly all of the forwarder's time can be spent waiting. In AI Gateway's first-month analysis, only 1,200 of roughly 16,000 runtime hours were CPU work, and the rest went to waiting on providers. Hosting billed by wall-clock time charges for that waiting time. Fluid compute pauses Active CPU billing whenever a function waits on input/output (I/O), but provisioned memory remains billable during those waits.
Copy link to headingA key store that keeps provider keys private
The key store holds provider keys that let the proxy call model services and bill your provider account. Proxies with client-key management also issue keys that let your app and teammates call the proxy, with model restrictions and spending limits where configured.
An admin key needs particular care. A self-hosted proxy with key management may have an admin key that can create and revoke client keys, so whoever holds it controls that access. Store it in a secret manager, keep it out of source control, and rotate it after any suspected leak.
Copy link to headingWhen an LLM proxy is enough and when to move to a gateway
A plain proxy can handle basic forwarding for a first LLM feature. A managed gateway can also provide centralized authentication, usage visibility, and provider failover from the first model call. Further requirements can arise as the app adds providers, teammates, and traffic.
Copy link to headingStay on a proxy while you call one provider
A server-side route or the provider's own SDK can handle basic forwarding for one provider. Consider a managed gateway when you want centralized authentication, usage visibility, or provider failover, even for a single model. Spend control can come from the provider's own spending limits, since a plain proxy has no budget store. Check whether those limits enforce a cap or only send alerts. OpenAI's spend settings distinguish the two. Tracing tools work with direct SDK calls too, while a gateway can centralize request usage and routing visibility. Provider count alone doesn't determine when to add one.
Copy link to headingPrice the self-hosting stack before you commit
A stateless proxy still depends on stateful infrastructure once it handles keys and spend. LiteLLM's production guidance recommends two or more replicas behind a load balancer and PostgreSQL for database-backed authentication and spend tracking. Redis coordinates shared state across workers and replicas, and database-backed deployments need schema migrations when upgrading. If self-hosting still fits those requirements, compare open source gateways by operating workload rather than feature lists.
Copy link to headingPatch the proxy like a security boundary
Self-hosting gives you control over deployment and configuration. Your team also owns upgrades, credential management, scaling, and incident response. Treat the proxy as a security boundary: restrict administrative access, protect stored credentials, and plan how to respond if a key or deployment is compromised.
Keeping a self-hosted proxy safe means pinning verified versions, applying security updates, watching security advisories, and rotating provider keys whenever exposure is suspected. Your team owns that work for as long as the proxy runs.
Copy link to headingAdopt a gateway when a second provider arrives
A second provider can turn direct SDK calls into a routing problem. Routes that choose among providers need to know which one serves a request and what happens when that request fails. Retry code can differ per provider. Managed AI gateways move that logic from app code into one routing layer. After Zo Computer moved retries and fallback into AI Gateway, its retry rate fell from 7.5% to 0.34%. With the AI SDK and AI Gateway together, adding a new model went from an hour-long, multi-file change to adding a config string in 30 seconds.
Copy link to headingMeasure hop latency against real providers
A separate proxy adds a network hop. Overhead figures measured against a mock provider with logging turned off can rank implementations, but they won't predict production response times. Measure latency against your real providers with streaming and logging enabled.
Copy link to headingHow AI Gateway handles LLM proxying on Vercel
Vercel AI Gateway provides unified model access through a managed API, with provider routing, spend controls, usage tracking, and billing in one place. It works with applications hosted anywhere through the AI SDK or compatible OpenAI clients, with no proxy servers or database to run. On Vercel, the AI SDK can also authenticate automatically with short-lived OpenID Connect (OIDC) tokens.
Copy link to headingKeep provider keys out of your app with OIDC
Copying a provider key into preview, production, and every laptop can mean three or more copies to rotate. Each copy is a place it can leak. On Vercel, your app signs in to AI Gateway with a short-lived OIDC token. The gateway attaches the provider credentials, so your code never needs to handle them.
The AI SDK uses the token automatically on Vercel deployments as long as no explicit API key or AI_GATEWAY_API_KEY is set. Locally, rerun vercel env pull to refresh it once its 12-hour lifetime expires, as described in the OIDC documentation. Outside Vercel, an AI Gateway API key with an expiry date authenticates requests instead and stays server-side like any other secret.
Copy link to headingKeep requests running when a provider fails
A provider outage or a context-limit error can show up in your app as a failed chat. AI Gateway picks a provider based on recent uptime and latency, and providerOptions.gateway lets you set your own provider order plus a list of fallback models. Errors including context limits, unsupported inputs, and outages can trigger model fallbacks, though a request still fails if no eligible route succeeds.
This Next.js Route Handler prioritizes Azure then OpenAI for the requested model and specifies two backup models. The order setting expresses a preference rather than restricting routing to those providers. Install ai with pnpm add ai, follow the OIDC setup, and place the handler in app/api/chat/route.ts:
import { streamText } from 'ai';
export async function POST(req: Request) { const { prompt } = await req.json();
const result = streamText({ model: 'openai/gpt-6.1-sol', prompt, providerOptions: { gateway: { order: ['azure', 'openai'], models: ['anthropic/claude-opus-5.5', 'openai/gpt-6-luna'], }, }, });
return result.toTextStreamResponse();}The handler doesn't pass an API key because the AI SDK handles OIDC on Vercel. In production A/B testing against its previous infrastructure, Cline reported 10–14% lower 99th-percentile (P99) streaming latency across its most-used models and 24.3% fewer errors.
Copy link to headingCap spend per team, project, key, or person
A retry loop or an overshared key can generate charges before anyone checks the dashboard. AI Gateway budgets set a spending cap for a team, project, API key, or team member. The gateway checks applicable budgets before each request. A request that exceeds the budget can be completed. The gateway rejects subsequent requests with HTTP 402 until the budget resets or someone raises it.
Budgets are soft caps on spend through AI Gateway's own credentials, so traffic billed to your own provider keys still needs the provider's limits. A key with no budget in scope has no cap. Set limits before you share access.
Team-member budgets cover API keys attributed to that member. OIDC-authenticated deployment traffic counts toward project and team budgets; limits for individual application users require application-level controls.
Copy link to headingSee every routing attempt in request logs
A plain proxy needs logging to help investigate slow or failed requests. The AI Gateway Logs page lists every request with its cost, tokens, and latency. For a fallback, it shows each provider attempt in order with its status. These routing details stay available for 30 days.
AI Gateway records this metadata without retaining your prompts or responses after the request completes. Upstream provider retention depends on the provider and your data-retention settings. Teams on Pro and Enterprise that already run an observability tool can export request metadata through Trace Drains. These send OpenTelemetry traces according to configured sampling rules and incur trace-event and egress charges.
Copy link to headingShip LLM features without running your own gateway
Production LLM traffic often needs more than a forwarded request. It may need a fallback when a provider errors, a cap on what each caller can spend, and a record of what happened to every call. A minimal forwarding proxy leaves each of those for your team to add. Whether that work is worthwhile depends on your providers, your team, and your traffic.
AI Gateway provides those controls as a managed layer your app configures:
OIDC authentication: Deployments sign in with short-lived tokens, and the gateway attaches provider credentials outside your code.
Provider and model fallbacks: A provider order and a list of backup models let requests try eligible alternatives after provider errors.
Budgets at four scopes: Configure soft spending caps across teams, projects, API keys, and team members.
Request logs: Each request shows its cost, tokens, and the provider attempts behind it.
OpenAI-compatible API: Compatible OpenAI clients can migrate by updating the base URL, authentication, and model ID. Check feature support for the API they use.
Follow the AI Gateway getting started guide to route your first request from a new or existing application, or browse templates for an AI chatbot starter.
Copy link to headingFrequently asked questions about LLM proxies
Copy link to headingIs an LLM gateway the same as an LLM proxy?
Not always. The names get used interchangeably, so the feature list is the better test. A tool that only forwards requests is working as a minimal proxy. One that also retries failed requests, enforces budgets, and logs each call provides gateway capabilities, even if it is called a proxy. The LLM gateway guide explains these capabilities in depth.
Copy link to headingIs LiteLLM a proxy or a gateway?
Both, depending on how you run it. LiteLLM ships a Python SDK for calling providers directly from your code and a self-hosted server that adds virtual keys, budgets, and logging. Using its database-backed key management and spend tracking also means running a database and keeping up with security releases. The LiteLLM alternatives guide covers this tradeoff.
Copy link to headingDoes an LLM proxy slow down responses?
A separate proxy adds a network hop. Its effect depends on deployment and routing, and the provider's response time usually determines how long a streamed reply takes. On AI Gateway, the Logs page shows time to first token for each provider attempt, so you can investigate whether a slow request came from the provider or the routing.