Pick a gateway off a feature checklist and the pick will usually be wrong. Teams compare per-token pricing across a spreadsheet, pick the cheapest row, and ship it. That approach stopped working the moment every major gateway here (Vercel, OpenRouter, Portkey, LiteLLM, Cloudflare, Kong, and Helicone) dropped its per-token markup to zero. The race to the bottom on price is over. What's left to evaluate is deployment model and failover behavior, and most teams never get around to testing either before they're in production.
We operate one of these seven gateways, and we've watched this play out from the routing layer during weeks that weren't quiet. In December 2025, OpenAI logged 22 incidents totaling 182.7 hours of impact, and Anthropic logged 20 incidents totaling 184.5 hours, in a single month. That's not an edge case anymore. Thirty-seven percent of teams now run five or more models in production, up from 29% the year before, which means a single-provider integration has already become the exception. A gateway's whole job during an incident like that is deciding, in milliseconds, whether the application notices.
Most teams find out how their gateway handles that decision after the first outage, not before. That's backwards. Failover behavior is testable against real traffic in an afternoon, and the answer tells you more about whether a gateway fits your stack than any feature list will.
Copy link to headingKey takeaways
Per-token markup has gone to zero across every major gateway in 2026; the differentiators now are deployment model and failover behavior, with observability depth close behind.
Automatic failover varies by gateway. Cloudflare AI Gateway retries transient errors automatically but still requires Dynamic Routing to fail over across providers.
Our production index shows fallback rescuing 3.5% of requests and 5.1% of tokens, over one trillion tokens per month that would otherwise have returned errors.
Most published gateway latency benchmarks test against mock upstreams, and with LLM responses taking 500ms-3s, sub-50ms gateway overhead sits inside the noise.
Deployment context (Vercel-native, Cloudflare-native, Python stack, air-gapped) determines the right gateway more than feature count does.
Copy link to headingWhat an AI gateway handles that application code shouldn't
An AI gateway sits between the application and the provider APIs, handling the token-based budgets, multi-provider routing and failover, semantic caching, data-residency enforcement, and per-request observability that no team should be rebuilding per project. It functions as a control plane for LLM usage, and Gartner named the category an emerging one in its 2024 API Management Hype Cycle.
Skip the gateway and each team ends up managing its own provider credentials, writing its own retry logic, and logging to its own place. Reliability is a platform concern, not something every team should rebuild for itself. In our experience, teams that skip this layer don't save the work. They defer it until the first incident forces the build under pressure, with no time to get the retry semantics right.
The traffic pattern makes this worse, not better. Agent-driven workflows now generate more than half of all output tokens, and agents don't pause to notice a provider is down the way a person would. A single-provider API call stopped being an acceptable production architecture around the same time agents started outnumbering chat sessions in the traffic mix.
Copy link to headingHow we evaluated these seven AI gateways
Deployment model, failover default behavior, latency in the context of an actual LLM call, pricing structure, and self-hostability drove this comparison, weighted in that order.
Deployment model decides where the gateway runs and what that means for data residency and operational overhead.
Failover default determines whether provider errors get handled inside the gateway automatically or require configuration first.
Latency in context measures overhead relative to a full LLM response rather than in isolation.
Pricing structure shows how the gateway makes money now that token markup is gone everywhere.
Self-hostability determines whether running the gateway inside your own infrastructure is an option at all.
Copy link to headingDeployment model
Deployment model comes first because it decides where data travels and how much operational surface a team owns going forward. A managed gateway is one less thing to run. A self-hosted one keeps everything inside your own network, at the cost of servers, monitoring, and someone on call for it.
Copy link to headingFailover default
Failover default came right after that. Not simply whether a gateway can handle a failed request somehow, but whether it does so by default or needs configuration a team might not think to add until the first outage. That split is the single biggest one on this list. Cloudflare AI Gateway shows it clearly, since transient retries now happen automatically but cross-provider failover still doesn't.
Copy link to headingLatency and pricing
We treated latency the way we'd treat it in a postmortem, relative to the whole request rather than in isolation. Forwarding overhead measured in single-digit milliseconds barely registers against an LLM call that takes half a second to three seconds to complete. Price landed fourth for the same reason it opened this piece. It stopped being a useful lens the moment every gateway here converged on zero markup.
Copy link to headingThe seven gateways
Every entry below covers what the gateway is, who it fits best, and where the fit runs out, in that order. The order follows deployment model, from fully managed through fully self-hosted, since that's the axis most likely to rule an option in or out before anything else does.
Copy link to headingVercel AI Gateway
Vercel AI Gateway went into alpha in May 2025, building on what we'd learned scaling v0 across multiple model providers, and reached general availability three months later. It runs as a Node.js service on our own infrastructure across multiple AWS regions on Fluid compute, and nothing on the request path leaves Vercel's network. AI SDK 5 and later points at it by default, so a plain model string routes through the gateway with no added configuration, across 200+ models from 40+ providers.
Best for: teams already deployed on Vercel or building with AI SDK, since the integration costs nothing beyond a model string, and OIDC handles authentication without a separate secrets story.
Worth knowing before you commit: it isn't self-hostable. If an architecture requires the gateway to run inside a team's own network boundary rather than ours, this isn't the option, regardless of how it performs everywhere else.
Copy link to headingOpenRouter
OpenRouter is managed only, raised a $113M Series B in May 2026, and carries the broadest catalog on this list at 400+ models across 70+ providers. It charges no per-token markup. Revenue comes from a 5.5% fee on credit purchases, and failover is automatic, with routing that accounts for both quality and cost.
Best for: teams that want the widest model selection without running infrastructure themselves, and that don't mind the credit-purchase fee as the price of that breadth.
Worth knowing before you commit: there's no self-hosting option, and the 5.5% fee applies regardless of usage pattern, which shows up more at volume than it does during testing.
Copy link to headingPortkey
Portkey ships as managed SaaS, starting at $49 a month for its Production tier, or as a self-hosted open-source gateway, and covers 1,600+ providers and models, the largest catalog on this list. Palo Alto Networks acquired the company in May 2026. It's SOC 2 certified, ships automatic fallbacks on every plan, and its dashboard covers traces, errors, caching, and cost together.
Best for: teams that want the option to start managed and move to self-hosted later without switching gateways, or that need the widest provider catalog paired with day-one automatic failover.
Worth knowing before you commit: the acquisition is recent enough that roadmap direction under new ownership is still an open question, not a settled one.
Copy link to headingLiteLLM
LiteLLM is an MIT-licensed, self-hosted proxy supporting 100+ providers, deployable on Kubernetes, Docker, or a single VM, including air-gapped environments. Netflix, Stripe, and federal agencies all run it in production. SSO, RBAC, and team-level budget enforcement sit behind the paid Enterprise tier.
Best for: regulated or air-gapped environments where nothing can leave a team's own infrastructure except the LLM API call itself, and where the operational cost of self-hosting is already budgeted for.
Worth knowing before you commit: failover isn't automatic out of the box, same as everything else the managed gateways here give by default, which means the real cost of LiteLLM shows up in the infrastructure around it, rather than the license. Self-hosting budgets typically run $500 to $2,000-plus a month once servers, Redis, Postgres, and monitoring are counted, based on published cost breakdowns.
Copy link to headingCloudflare AI Gateway
Cloudflare AI Gateway runs managed and edge-distributed across 300+ points of presence, with core features (analytics, caching, rate limiting) free on every plan. Failover here has two layers, and the distinction matters. Since April 2026, the gateway retries transient upstream errors automatically, up to five attempts with configurable backoff, without any client-side changes. Failing over to a different provider is a separate thing entirely, and that still requires Dynamic Routing configured explicitly.
Best for: teams already built on Cloudflare's edge network who want gateway features as a one-line addition, and who are prepared to configure Dynamic Routing for cross-provider coverage rather than assuming the automatic retries cover it.
Worth knowing before you commit: the automatic retries handle transient failures, but they're not the same thing as failing over to another provider. Skip the Dynamic Routing setup and a genuine provider outage still reaches the application the way it would without a gateway at all.
Copy link to headingKong AI Gateway
Kong AI Gateway is open-source and self-hostable, or available through Kong Konnect as SaaS, priced roughly $500 to $2,500 a month, and SOC 2 certified. Semantic routing, which routes requests by similarity between the prompt and a model's description rather than by static rules, has been in the product since version 3.8. The current release, 3.14, extends that with GA agent-to-agent governance and a broader MCP feature set, including MCP Tool ACLs and a Kong MCP Registry, for teams running agent infrastructure through the same gateway.
Best for: teams already running Kong for API management who want AI routing inside the same platform, particularly where semantic routing or agent-to-agent traffic governance is already a requirement.
Worth knowing before you commit: it's the most expensive option here by a wide margin at the Konnect tier, and that cost only makes sense for teams already inside the Kong ecosystem rather than adopting it purely for AI gateway features.
Copy link to headingHelicone
Helicone is open-source and self-hostable, with its AI Gateway launched in June 2025. It charges zero markup on inference and reports the lowest latency overhead on this list, 1-5ms at P95, though the project is currently in maintenance mode under Mintlify.
Best for: prototypes and low-volume, cost-sensitive workloads where the lowest reported overhead and zero markup matter more than active roadmap development.
Worth knowing before you commit: the maintenance-mode status matters more than any latency figure once production traffic is actually on the line.
Copy link to headingWhere the seven gateways diverge
Each gateway above stands on its own, but the differences are easiest to weigh side by side. Two columns do most of the deciding for most teams: deployment model and failover default. Managed gateways, Vercel AI Gateway included, hand the operational overhead off entirely, which is exactly what a team wants once it's decided it doesn't need to run the gateway itself. Teams that do need to self-host for data residency or compliance reasons have Portkey, LiteLLM, Kong, and Helicone as options. The failover column matters just as much, since it separates the gateways that handle failover automatically from the ones that need configuration first. Latency, pricing, catalog size, and observability are real differences, but they're tiebreakers once those two constraints have narrowed the field.
The failover column changes the operational contract most. For example, Cloudflare AI Gateway now retries transient errors on its own, but a genuine provider outage still needs Dynamic Routing configured before it behaves like the other managed options in the table.
Among the managed gateways in this comparison, only Vercel AI Gateway exposes a live REST endpoint reporting P50 and P95 latency and throughput per model and provider, refreshed hourly and queryable before a request routes at all.
Copy link to headingLatency benchmarks measure the wrong thing
Published gateway benchmarks tend to isolate forwarding overhead, which is a strange thing to isolate, given how little of the request it represents. Reported overhead figures span five orders of magnitude across sources, largely because most tests measure forwarding against a mock upstream instead of a real provider. That approach hides the variable that actually dominates a request, which is how long the provider itself takes to respond. For LLM calls running 500ms to 3 seconds, the gateway overhead sits well inside the noise.
Our own production data makes the point sharper. In AI Gateway's first month of traffic, total runtime came to roughly 16,000 hours, and only 1,200 of those hours went to real CPU work. The other 14,800 hours were idle, waiting on provider responses, putting the split at roughly 92% waiting and 8% computing. An AI gateway is a network-bound workload before it's anything else, so forwarding latency only ever touches that 8% slice.
Rescue rate is where uptime actually lives. Through April 2026, the same fallback path behind the 3.5% request figure rescued 5.1% of all tokens and 4.9% of market cost, which at this scale works out to over one trillion tokens recovered per month that would otherwise have returned errors. Vercel is the only gateway in this comparison with published, equivalent production rescue-rate data.
The practical heuristic holds regardless of which gateway is under evaluation. If calls average 1,000ms, the difference between a 20ms and a 5ms gateway works out to a 1.5% latency delta. A 3.5% request rescue rate is a 3.5% uptime delta. One of those numbers is the one that ends up on an incident report.
Copy link to headingTest the failure before the feature list
Every gateway on this list has converged on price and diverged on operational behavior. The divergence that actually bites is what happens when a provider returns a 500 at two in the morning, and that's testable before anything else here, not after.
The deployment context decides the answer faster than any feature comparison:
Vercel-native or AI SDK apps: Vercel AI Gateway, since OIDC authentication removes secrets management entirely, and a plain model string routes through with no added infrastructure
Multi-cloud apps needing the broadest catalog: OpenRouter for quality-aware routing across 400+ models, or Portkey for 1,600+ providers under SOC 2
Regulated or air-gapped environments: self-hosted LiteLLM, budgeting for the infrastructure around it rather than a license fee
Existing Cloudflare infrastructure: Cloudflare AI Gateway as a one-line addition, with Dynamic Routing configured explicitly for cross-provider failover rather than relying on automatic retries alone
Kong-native API platforms: Kong AI Gateway, particularly where semantic routing or agent-to-agent traffic governance is already a requirement
Prototypes and low-volume, cost-sensitive workloads: Helicone, with the maintenance-mode status, weighed against the lowest reported latency here
None of that testing has to be expensive. Vercel's free tier includes $5 in monthly credits, enough to point an AI SDK model string at the gateway and watch the provider metadata on each response to confirm failover does what the documentation claims.
The question was never which gateway has the longest feature list. It's what happens to a request the moment a provider stops responding, and every gateway here answers that question differently, whether a team has tested for it or not.
Copy link to headingFAQ
Do any of these gateways add a per-token markup in 2026?
No. All seven pass provider token rates through unchanged and make money elsewhere. OpenRouter takes a 5.5% fee on credit purchases, Portkey starts at $49 a month, and LiteLLM shifts the cost to a team's own infrastructure instead of a markup. Vercel AI Gateway charges zero markup, including on BYOK requests on the paid tier.
Is Cloudflare AI Gateway's failover automatic?
Partially. Since April 2026, Cloudflare AI Gateway retries transient upstream errors automatically, no configuration required. Failing over to a different provider is a separate capability, and Dynamic Routing has to be configured before that happens. Vercel AI Gateway, OpenRouter, Portkey, and LiteLLM (once configured) all handle cross-provider failover at the gateway layer before the application sees a response.
Can Vercel AI Gateway be used with frameworks other than Next.js?
Yes. It works with AI SDK 5 and later, including AI SDK 7, and it exposes an OpenAI-compatible API (Chat Completions and Responses) plus an Anthropic Messages API. Anything able to send an HTTP request to https://ai-gateway.vercel.sh/v1 can route through it.
What does "zero markup including BYOK" mean?
Bring Your Own Key routes requests through a team's own provider credentials at provider list rates, and the gateway adds nothing on top. Cost and token data still get captured alongside request data, the same way they do for gateway-billed traffic. BYOK is available on the paid tier only.
How do you know if a gateway's latency overhead matters for a given workload?
Compare it against the model's actual response time. For calls lasting 800ms to 2 seconds, the difference between a 10ms and a 40ms gateway is under 4% of total latency. The rescue rate on failed provider requests moves uptime by a wider margin than that gap ever will.