Picking an open source AI gateway comes down to how it behaves under load, what it does when a provider goes down, what its license covers, and how much it takes to run. This guide compares LiteLLM, Portkey, Envoy AI Gateway, and Bifrost, then shows when a managed gateway like Vercel AI Gateway is the better fit.
Key takeaways:
LiteLLM: The widest provider support and the fastest path to a working proxy, though its Python runtime slows down under heavy load.
Portkey: MIT-licensed, with fallbacks, retries, and guardrails in the open source core.
Envoy AI Gateway: Built on Envoy under Apache 2.0, and the pick for teams already on Kubernetes.
Bifrost: A Go gateway that advertises sub-millisecond overhead, though independent runs disagree, so test it on your own traffic.
Managed gateway versus self-hosting: Running a gateway yourself trades away the setup, scaling, and patching work that a managed layer absorbs.
Copy link to headingWhat is an open source AI gateway?
An open source AI gateway is a proxy you host yourself, sitting between your app and the large language model (LLM) providers you call. It gives every call one API, with shared authentication, routing, failover, and spend tracking. Without it, you wire each provider's SDK into your code and handle rate limits, retries, and key rotation for every one. With a gateway, you point your code at a single endpoint and configure providers once.
Open source means it runs on your own infrastructure, under a license that lets you self-host and modify it. Open source gateways provide more control, yet they become one more service to deploy, scale, secure, and patch.
Copy link to headingOpen source AI gateways at a glance
The four gateways here cover most of the open source shortlist in mid-2026. They split on two aspects. The first is the runtime each is built on, which caps how much traffic it takes. The second is how much of the feature set lives in the open source core.
The table below maps each gateway to the factors that inform production fit.
The breakdowns below start with the gateways most teams try first.
Copy link to headingLiteLLM
LiteLLM is an MIT-licensed Python proxy that routes to more than 100 providers through one OpenAI-compatible API. It's the most popular gateway here based on GitHub stars. For a solo developer or a small team, it's the fastest way to stand up a proxy with budgets, load balancing, and cost tracking already working.
Key features:
Provider breadth: More than 100 providers behind one OpenAI-compatible API, with virtual keys for per-team budgets.
Cost tracking and load balancing: Built-in spend logs and load balancing across keys and deployments.
Self-hosted or hosted: Run it yourself, or use the maintainer's managed version.
Pros:
Fastest to a working proxy: The OpenAI-compatible API drops into existing code with a base-URL change.
Widest provider coverage: The broadest catalog here, all behind one API.
Active development: Frequent releases and a large contributor base.
Cons:
Python slows down under load: Below a certain concurrency, overhead stays in the low single-digit milliseconds, but throughput can drop sharply under heavy load, and tuning database pooling or file limits doesn't reliably recover it.
Identity sits outside the core: Single sign-on (SSO), OpenID Connect (OIDC), and SCIM (System for Cross-domain Identity Management) aren't in the MIT core.
A 2026 supply-chain incident: Two PyPI releases briefly shipped malicious code that went after cloud credentials and secrets. It's one of the most-pulled packages in its category, which is what made it a target.
Best for: Solo developers and small teams after the widest provider support and the fastest setup, as long as traffic stays inside what the Python runtime handles comfortably. The core is MIT-licensed, and the identity features sit outside it.
Copy link to headingPortkey
Portkey is an MIT-licensed gateway built on Node.js that routes to a large model catalog. It open-sourced its production gateway, so fallbacks, retries, load balancing, and guardrails now run self-hosted.
Key features:
Fallbacks and retries: Circuit-breaker fallbacks across providers, plus automatic retries with backoff.
Guardrails in the core: Input and output checks callable through the same API.
Multimodal routing: Vision, audio, and image models through one OpenAI-style API.
Pros:
Reliability in the core: Fallbacks, retries, timeouts, and load balancing ship in the MIT core.
Broad model catalog: One API across a large set of providers and models.
Guardrails built in: Safety checks run in the same path as routing.
Cons:
Concurrency is less documented: Published benchmarks are thin, so its throughput ceiling is harder to reason about than LiteLLM's or Bifrost's. You'll want to measure it on your own workload.
Governance sits outside the core: Longer log retention and compliance attestations live in the hosted version.
Best for: Production reliability without wiring fallbacks, retries, and guardrails together yourself. The core is MIT-licensed, with retention and compliance features in the hosted version.
Copy link to headingEnvoy AI Gateway
Envoy AI Gateway is an Apache 2.0 project built on Envoy Proxy, with a Go control plane driving Envoy's C++ data plane. It adds model-aware routing, failover, and upstream authorization to a proxy platform teams already run. If you've already got Kubernetes and a service mesh, it extends what you have instead of adding a new service.
Key features:
Envoy-native routing: Model traffic management on top of Envoy Gateway, with rate limiting and policy control.
Failover and upstream auth: Failover across providers, plus end-to-end authorization for model traffic.
MCP and multimodal: Model Context Protocol (MCP) support and multimodal routing.
Pros:
Fits an existing Envoy stack: Reuses the proxy, policy, and observability you already run.
Fully open under Apache 2.0: The routing core isn't behind a separate license.
Community-backed: Developed in the open by the Envoy community, with regular releases.
Cons:
Heavier to run: Running it means running Envoy, which is more infrastructure than a single-binary proxy for a small team.
Smaller ecosystem: Its community and third-party docs are thinner than the most popular proxies, so edge cases mean reading source.
Best for: Platform teams already running Envoy or a Kubernetes service mesh, where model routing can live inside the mesh. Fully open under Apache 2.0, with support available through Envoy vendors.
Copy link to headingBifrost
Bifrost is an Apache 2.0 gateway written in Go, competing on raw speed. It advertises sub-millisecond routing overhead at high request rates. Independent testing muddies that picture.
Key features:
Go runtime built for throughput: A compiled, concurrent runtime aimed at high request rates.
Semantic caching: Response caching backed by a vector store (Weaviate).
Ungated core: Core routing features under Apache 2.0.
Pros:
Low overhead by design: A Go runtime keeps per-request cost small at high request rates.
Open core: The routing features most teams need come without a separate license.
Cons:
Benchmarks disagree: Its own numbers show minimal overhead, but independent runs have seen it fall over under high concurrency, and the results hinge on tuning. The safe move is to benchmark it on your own traffic before trusting a headline figure.
An enterprise tier is forming: Guardrail chaining across services like Bedrock and Azure Content Safety already lives in a separate enterprise build.
Best for: High-concurrency Go teams after minimal overhead, once they've benchmarked it against their own traffic. Fully open under Apache 2.0, with an enterprise tier taking shape.
Across the four, the call is less about a single winner and more about which runtime and license fit your team.
Copy link to headingHow to choose an open source AI gateway
The shortlist shrinks fast once you're honest about three things. The first is how much traffic this will see. The second is how much operational work the team can absorb. The third is which identity, governance, or compliance features you need on day one. The table maps common profiles to a starting point.
These are starting points, not verdicts. The filters below are where the call gets made.
Copy link to headingRuntime and concurrency needs
Runtime is the first filter, since it caps how much traffic the gateway takes. LiteLLM's Python proxy is the fastest to stand up and supports the most providers, and its overhead stays low at moderate load. The drop-off under heavy concurrency is what to watch, because a gateway that's a convenience at 100 requests per second can become the bottleneck well before production peak. Go-based options like Bifrost and Envoy AI Gateway handle higher rates, though they're less forgiving to set up.
Copy link to headingSelf-hosting control versus operational cost
Self-hosting buys full control of the request path, and you pay for it in operational work. Spend tracking usually puts a database like PostgreSQL in the path of every call, so a slow query slows every model call. Connection pooling, high-availability deployment, and dependency auditing all land on the team. The 2026 LiteLLM incident is the sharp version of that last point, since a self-hosted gateway is only as safe as every tool in its build and release pipeline. That's a real cost of self-hosting, not a reason to avoid it. What it comes down to is whether you'd rather own that work or hand it off.
Copy link to headingKubernetes and infrastructure fit
If you already run a service mesh, the gateway should extend it, not sit beside it. Envoy AI Gateway fits here, since it builds on Envoy and reuses your existing policy and observability. Without that setup, there's little reason to bring in Envoy for model routing alone, and a single-binary proxy is usually the better call.
Copy link to headingCompliance and data retention
Compliance can settle the shortlist before performance does. In most open source gateways, single sign-on, audit logs, and data-retention guarantees sit outside the core. A team that needs them on day one can end up running a separate edition anyway. When Zero Data Retention or signed compliance attestations are hard requirements, the build-versus-buy math tips toward a managed layer that includes them, which is where a managed gateway changes the picture.
Copy link to headingHow Vercel AI Gateway compares on ops, edge, and reliability
Vercel AI Gateway is managed rather than self-hosted, so the comparison shifts, weighing control against work you no longer have to do. It's one OpenAI-compatible endpoint for hundreds of models across dozens of providers, and the gains show up where self-hosting struggles most, in reliability, edge performance, and operations.
Copy link to headingProvider outages break single-provider apps
A gateway's main job is to absorb provider trouble. When a provider errors, throttles, or times out, an app wired straight to it goes down too. Provider outages were a recurring story across 2025\.
Vercel AI Gateway runs circuit breakers, retries within a model group, and cross-provider fallback. Every response carries metadata showing which provider served it, any fallback attempts, latency, and token cost. Across production traffic through April 2026, automatic fallback rescued 3.5% of requests, with the token and cost rates running higher because the rescued calls are the bigger ones. After moving its traffic over, the coding agent Cline saw streaming tail latency improve and error rates drop 43.8%.
Copy link to headingGlobal routing keeps latency low at the edge
Self-hosted gateways usually run in one region. A request far from that region pays the round trip twice, once to reach the gateway and once to reach the provider. Vercel AI Gateway runs on the same global network that fronts Vercel, across 126 points of presence and 20 compute regions. Requests enter close to their origin and cross a private backbone to the nearest region, holding sub-20ms overhead on model calls.
Copy link to headingSelf-hosted routing adds operational cost
Running a gateway means running everything under it. That's the database in the spend-tracking path, the cache, a high-availability deployment, and the on-call rotation behind all of it. Vercel AI Gateway takes that layer off the team, with no stateful services to operate.
Okara, which runs marketing agents for 120,000 companies, swapped every custom provider integration for one configuration and moved retry and fallback logic into the routing layer. A new model goes live for them without a new adapter or another test-and-deploy cycle.
Copy link to headingKey management and compliance guarantees
Every provider you add is one more key to store, rotate, and keep out of source control. Every regulated workload is one more data guarantee to prove. On Vercel, OIDC tokens are provisioned automatically, so there are no gateway keys to manage in production. Bring Your Own Key (BYOK) requests get a backstop. If a call on your own credentials fails, the gateway retries with system credentials instead of dropping the request. Zero Data Retention routing covers workloads that can't store prompt or completion data, a feature most open source gateways leave you to build yourself.
Copy link to headingChoose between self-hosted and managed routing
A gateway exists to absorb provider trouble while keeping its own failures small. The decision comes down to which failure you'd rather own. Self-hosting with LiteLLM, Portkey, Envoy AI Gateway, or Bifrost makes sense when you need full control of the request path, the team can run the stateful pieces, or a rule keeps every component in-house. A managed layer makes sense when reliability, edge performance, and operations should belong to the platform instead of the team.
For teams already on Vercel, Vercel AI Gateway handles that operational layer:
Automatic failover with observable routing: Circuit breakers, retries, and cross-provider fallback, with per-request metadata on provider, latency, and cost.
OIDC credential injection: Automatic token provisioning for apps on Vercel, with no gateway keys to rotate.
Zero Data Retention routing: Compliance-grade routing for workloads that can't store prompt or completion data.
Global network routing: 126 points of presence and 20 compute regions, with sub-20ms overhead on model calls.
Managed operations: No database, cache, or high-availability deployment to run yourself.
Start a new project and point your app at Vercel AI Gateway, or browse templates for a working setup to build on.
Copy link to headingFrequently asked questions about open source AI gateways
Copy link to headingIs Portkey's gateway fully open source?
Yes. Its production gateway is open source under the MIT license, including fallbacks, retries, load balancing, and guardrails. Longer log retention and compliance attestations live in the hosted version, but the routing gateway itself is open source and self-hostable.
Copy link to headingIs Envoy AI Gateway production-ready?
It's built on Envoy Proxy under Apache 2.0 and actively developed by the Envoy community. It fits platform teams already running Envoy or a Kubernetes service mesh. Teams without that setup take on real operational weight to adopt it.
Copy link to headingIs LiteLLM safe to run after the 2026 supply-chain incident?
Two PyPI releases were affected. The fix is to pin verified releases, audit the dependency tree, and watch upstream packaging tools. The underlying exposure, a high-download Python package with a large dependency surface, outlasts the one incident.
Copy link to headingWhen does managed Vercel AI Gateway beat self-hosting?
When reliability, edge latency, and operations should belong to the platform. It fits teams on Vercel that want failover, OIDC key handling, and Zero Data Retention without running a database, cache, and high-availability setup themselves. Self-hosting wins when you need full request-path control.