Most self-hosting decisions for a large language model (LLM) get made with a cost calculator. Nearly all of those calculators ignore the variable that dominates the answer, which is how busy the GPU stays.

A managed API wins for most workloads, since the provider carries the idle capacity and the model upgrades. That changes under a residency mandate, an on-premise requirement, volume that saturates a cluster, or data-plane control, none of which turn on cost.

This guide covers what self-hosting commits a team to, where the cost comparison breaks down, and how to decide.

**Key takeaways:**

- Comparing your own GPUs against frontier API pricing overstates the case for self-hosting, because a team choosing open weights is weighing them against managed inference for those same weights.

- Cost per token is set by how busy the GPU stays, with published benchmarks measuring a 17.5 times swing on identical hardware between a near-idle GPU and a saturated one.

- Production model preference turns over faster than a self-hosted deployment cycle, with 81% of AI Gateway tokens in July 2026 running on models that were absent six months earlier.

- Residency and on-premise mandates are the conditions that make self-hosting an obligation rather than an optimization.

- The routing layer and the model layer are separate decisions, and settling routing first keeps every model choice reversible.

## [Copy link to heading](#what-is-a-self-hosted-llm)What is a self-hosted LLM?

A self-hosted LLM is an open-weight model running on GPU infrastructure a team controls, served by inference software that team also operates. The weights come from a public release such as the Llama, Qwen, Mistral, or DeepSeek families, downloaded and run directly rather than called through a provider's endpoint.

The difference from a managed API is where responsibility sits. Capacity planning, version upgrades, quantization choices, and failure recovery all become the team's work. Hardware can be rented as cloud GPUs or machines in a private rack, and that choice changes the shape of the bill without changing the operational load.

Where those responsibilities land depends on which layer a team takes on, and there are two of them.

## [Copy link to heading](#how-a-self-hosted-llm-deployment-works)How a self-hosted LLM deployment works

The model layer runs inference on GPUs. The routing layer above it decides where each request goes, and the two can be chosen independently.

### [Copy link to heading](#the-serving-engine)The serving engine

The serving engine loads the weights onto the GPU and turns incoming requests into tokens. It handles batching, key-value cache management, and request scheduling, which together set the throughput ceiling for a given model on a given hardware.

Modern engines batch requests continuously rather than in fixed groups, so throughput depends on how many requests are in flight at once. A GPU serving one request at a time produces a small fraction of the tokens it produces under concurrent load, on identical hardware, which is where most self-hosting cost estimates go wrong.

### [Copy link to heading](#the-routing-layer)The routing layer

The routing layer handles everything between the application and whichever model answers. The layer handles credential management, provider and model selection, fallback when a route fails, retention and residency policy, and spend visibility across providers.

This layer exists whether or not a team builds it deliberately. Applications that call providers directly still make routing decisions, scattered across services as per-provider SDK calls, retry logic, and API keys in environment variables. Consolidating them is what a gateway or proxy does.

Because the layers are independent, a team can self-host routing while calling managed model APIs beneath it, or run managed routing in front of its own models. Treating "self-hosted LLM" as one yes-or-no decision is what produces GPU purchases that a routing change would have solved.

## [Copy link to heading](#what-a-self-hosted-llm-costs-to-run)What a self-hosted LLM costs to run

Most comparisons put GPU spend next to frontier API pricing. This inflates the case for self-hosting. A team choosing open weights is weighing them against managed inference for those same weights, which ran at about [a seventh](/blog/deepseek-overtakes-google-on-volume-cost-per-token-falls) of frontier rates across [AI Gateway](https://vercel.com/docs/ai-gateway) in July 2026.

The larger error sits on the other side of the equation. Public calculators treat GPU utilization as a fixed input, set at 100%, when it's set by how many requests the GPU serves at once. A [concurrency benchmark](https://arxiv.org/abs/2606.11690) on H100 hardware found the same Mixtral 8x7B deployment costing $15.25 per million output tokens at one request per second and $0.87 at saturation, a 17.5 times swing on identical silicon. No single break-even token count exists, because the crossover follows the traffic pattern rather than the volume.

Here's how the two options differ across the axes that decide the outcome:

| Decision axis | Self-hosted | Managed API |
| --- | --- | --- |
| Upfront commitment | Rented GPU node billed hourly, or purchased hardware on a depreciation schedule | None |
| Dominant cost driver | Offered request rate, which sets the realized batch size | Tokens consumed |
| Idle behavior | Bills through idle hours at the full rate | Scales to zero between requests |
| Cost at low load | Up to 36.3 times the same hardware at saturation | Published per-token price |
| Cost at saturation | Approaches the calculator's figure | Published per-token price |
| Latency target | Tightening it caps sustainable load and raises cost per token | The provider's problem |
| Model update | Re-provision, re-validate, redeploy per cycle | Change a model string |
| Engineering load | Machine learning operations (MLOps) time from week one | None |

A service-level objective (SLO) is a cost decision as much as a reliability one. Tightening a tail-latency target caps the arrival rate a deployment can sustain before its tail breaches, which pushes the operating point above the cheapest place on the cost curve. The saturation prices quoted in vendor benchmarks are floors that an SLO-bound deployment never reaches.

None of these rules out self-hosting. It moves the decision off cost and onto the constraints that don't bend.

## [Copy link to heading](#four-cases-where-self-hosting-an-llm-is-the-right-call)Four cases where self-hosting an LLM is the right call

Each of these four overrides the cost comparison rather than winning it. None can be solved by changing how traffic is routed, which is why these cases are worth acting on.

### [Copy link to heading](#data-residency-and-sovereignty-mandates)Data residency and sovereignty mandates

Some workloads can't legally send a prompt across a border, which ends the cost conversation before it starts. The General Data Protection Regulation (GDPR) counts sending the prompt as processing, so the rule applies even when nothing is stored abroad. Financial services carry a further obligation under the Digital Operational Resilience Act, in force since January 2025, to state in their contracts where data sits and how they would leave a provider.

Managed routing covers a real part of this. [Regional inference](https://vercel.com/docs/ai-gateway/security-and-compliance/regional-inference) pins where a provider runs inference and stores anything it keeps at rest. A request that can't be honored in the pinned region fails with an HTTP 400 rather than running somewhere else. There is no silent cross-region fallback, which is the property an auditor asks about.

Pinning a region governs the provider, not the full network path, and abuse and safety review can fall outside the pinned region under a provider's own policies. A mandate that constrains only where inference and storage happen is satisfiable through managed routing. A mandate that constrains every hop the data takes is not.

### [Copy link to heading](#on-premise-and-private-cloud-mandates-in-regulated-industries)On-premise and private-cloud mandates in regulated industries

Healthcare draws the clearest line. Routing protected health information through an external API without an appropriate data processing agreement creates exposure under the Health Insurance Portability and Accountability Act (HIPAA), and consumer-grade endpoints fail that requirement by default.

A residency mandate asks where the data lands. A network-boundary mandate asks whether it left at all, and no amount of provider-side configuration answers that question, because the request still transits a network the team doesn't operate. Critical infrastructure operators and defense-adjacent workloads land here routinely.

Teams in this position should also check whether the mandate covers the model layer or the whole path. Where private networking satisfies it, [Secure Compute](https://vercel.com/docs/networking/secure-compute) provides dedicated private networks with virtual private cloud (VPC) peering, which resolves a class of these requirements without moving models onto owned GPUs.

### [Copy link to heading](#predictable-high-volume-inference-that-saturates-a-cluster)Predictable high-volume inference that saturates a cluster

A saturated GPU is cheap and an idle one is not, and the word doing the work there is sustained. Bursty traffic is the failure case, because a cluster bills through the troughs at full rate while a managed API scales to zero between spikes. The relevant number is not peak load but the arrival rate a deployment holds for months.

Narrow specialization strengthens this case more than raw volume does. Across a [study of 31 tasks](https://arxiv.org/abs/2405.00732), six of 10 fine-tuned models under 8B parameters outperformed GPT-4 on average, and all 10 beat GPT-3.5-Turbo, with each model fine-tuned on a single consumer GPU under 24 GB. The caveat is that those gains concentrated in traditional natural language processing tasks and were weaker on coding and mathematical reasoning.

That combination, a specialized model at sustained volume, is the strongest economic case for self-hosting and the least common.

### [Copy link to heading](#provider-neutrality-without-a-vendor's-control-plane)Provider neutrality without a vendor's control plane

Some teams cannot route prompts, completions, or audit records through any vendor's hosted control plane, regardless of retention policy. The architecture that fits is a control-plane and data-plane split, where a vendor manages policy and telemetry while inference runs inside the customer's own network.

This requirement concerns where requests travel, not where weights run, so it justifies self-hosting the routing layer. Self-hosting the models underneath requires its own separate justification, and teams that skip that step buy GPUs to solve a governance problem.

Each of these four holds against cost pressure. What none of them survives is the pace at which production model preference now moves.

## [Copy link to heading](#why-model-churn-favors-managed-apis-over-self-hosted-llm-deployments)Why model churn favors managed APIs over self-hosted LLM deployments

Self-hosting fixes a model at the moment of provisioning, and production preference no longer holds for that long. Across AI Gateway, [81% of July tokens](/blog/deepseek-overtakes-google-on-volume-cost-per-token-falls) ran on models that were not on the gateway six months earlier, while the open-weight share of volume climbed from 11% in April to 36% in July.

Every swap on a private cluster means re-provisioning, re-validating, and redeploying, against a target price that keeps falling while the hardware depreciates on a fixed schedule. On a managed API, the same swap is a configuration change, which is why teams at 10 million or more monthly requests run 35 distinct models across a [routing graph](https://vercel.com/i/llm-routing-strategies) rather than one deployment.

The same layer absorbs outages. Across seven months through April 2026, [fallback routing](/blog/ai-gateway-production-index) rescued 3.5% of all requests and 4.9% of market cost, and [Zo Computer](https://vercel.com/customers/how-zo-computer-improved-ai-reliability-20x-on-vercel) cut its retry rate from 7.5% to 0.34% after moving off direct provider APIs. Self-host on a concrete limitation, not a projection.

## [Copy link to heading](#managed-ai-gateway-vs.-self-hosted-llm-proxy)Managed AI gateway vs. self-hosted LLM proxy

A managed gateway operates the routing layer as a service. A self-hosted proxy, built on one of the [open-source gateway projects](https://vercel.com/i/open-source-ai-gateways), runs that same layer inside a team's own infrastructure, in front of managed model APIs, self-hosted models, or both.

The two options diverge across eight axes:

| Decision axis | Managed AI gateway | Self-hosted LLM proxy |
| --- | --- | --- |
| Setup | An API key and a base URL | Deploy the proxy and its supporting datastores |
| Infrastructure footprint | Operated for you | Proxy, database, and cache you run and patch |
| Network path | Requests transit the vendor's network | Stays inside your own VPC |
| Failover | Automatic across providers by uptime and latency | Fallback arrays you configure and maintain |
| Residency control | Region pinning where the provider supports it | Whatever your own network enforces |
| Air-gapped deployment | Not available | Supported |
| Credential handling | Bring your own key (BYOK) with no markup on the paid tier | Keys never leave your servers |
| New model availability | Available as providers add them | You integrate each provider yourself |

A mandate covering the network path, or an environment with no route to a public endpoint, rules out managed routing outright. Absent either, the proxy has to earn a platform team's time, because it becomes software running in the request path of every model call. The operational cost exceeds what the routing layer saves until spend is high enough to fund that team.

Everything else favors managed routing. [Okara](https://vercel.com/customers/how-okara-runs-cmo-agents-for-120000-companies-on-vercel) replaced its custom provider integrations with one AI Gateway configuration and moved retry and fallback logic out of its codebase, while running chief marketing officer agents for 120,000 companies.

## [Copy link to heading](#choosing-a-serving-engine-for-your-self-hosted-llm)Choosing a serving engine for your self-hosted LLM

If the four conditions hold and models are moving onto owned GPUs, the engine choice determines the cost floor. Three options cover most production deployments.

vLLM is the general-purpose default, with the broadest model and accelerator coverage and continuous batching that holds up under concurrent load. It's also the engine used in the concurrency benchmark, which makes its cost behavior the best documented of the three.

SGLang targets workloads with heavy prefix sharing. Its RadixAttention scheduler reuses shared key-value cache across requests, so retrieval-augmented generation, multi-turn chat, and agent loops that resend a large common context benefit most. The advantage narrows as prefix overlap falls.

Ollama gets a model running locally in minutes and is built for development rather than production traffic. Developers who benchmark it under realistic concurrent load generally move to vLLM before shipping. Measure at your own expected load before committing to a runtime.

## [Copy link to heading](#how-vercel-supports-self-hosted-and-managed-llm-deployments)How Vercel supports self-hosted and managed LLM deployments

Most of the pressure that pushes teams toward their own GPUs is about control, spend, and reliability rather than the weights themselves. AI Gateway addresses those at the routing layer, across 126 Points of Presence (PoPs) and 20 compute regions.

### [Copy link to heading](#pinning-inference-to-a-jurisdiction-without-moving-the-model-layer)Pinning inference to a jurisdiction without moving the model layer

A residency requirement is the first thing that turns a routing conversation into a hardware conversation, and it doesn't have to. Regional inference sets a US or EU region per request, and the provider runs inference and stores anything it retains in that region. A request that can't be honored there fails rather than routing elsewhere, and every response reports the region that served it, so residency can be verified rather than assumed. Bringing your own key doesn't opt a request out of that behavior. Region pinning governs where the provider runs and stores the request rather than the full network path, so a mandate covering every hop still points to your own infrastructure.

### [Copy link to heading](#comparing-build-versus-buy-against-the-provider's-real-rate)Comparing build versus buy against the provider's real rate

Cost comparisons break down when the routing layer takes a percentage, because the team ends up measuring its own GPUs against a marked-up price rather than the provider's. [AI Gateway pricing](https://vercel.com/docs/ai-gateway/pricing) applies no markup on tokens, including with bring your own key on the paid tier, so the managed number in a build-versus-buy model is the provider's published rate. Budgets cap spend at the team, project, or API key level, and the dashboard reports latency, token cost, and error rates across every provider in one place. Consolidating that view is work teams otherwise do themselves before they can run the comparison, and where [model spend control](https://vercel.com/i/llm-cost-management-track-control-model-spend) lives once the decision is made.

### [Copy link to heading](#constraining-which-providers-and-models-can-serve-a-request)Constraining which providers and models can serve a request

Compliance review approves a specific set of providers, and enforcing that approval across services is where the control argument for self-hosting starts. The provider allowlist and model allowlist are team-wide settings on Pro and Enterprise that reject a request with a 403 when no approved provider or model can serve it, and both apply to bring-your-own-key traffic. Zero data retention and disallow prompt training operate per request or team-wide, so retention posture is enforced at the routing layer instead of per service.

### [Copy link to heading](#absorbing-model-churn-instead-of-re-provisioning-for-it)Absorbing model churn instead of re-provisioning for it

Model churn is the recurring cost of owning the model layer, and it lands hardest on teams that validated a deployment once and expected it to hold. Switching models through AI Gateway is a one-line change, and routing rules let a team rewrite or deny a model across every request without touching application code. Fallback ordering across providers is configuration rather than software, so a provider outage redistributes traffic instead of surfacing as errors.

## [Copy link to heading](#settle-the-routing-layer-before-you-buy-gpus)Settle the routing layer before you buy GPUs

Self-hosting goes wrong when the decision rests on a cost number that assumes a utilization the traffic never produced. The constraints that genuinely require it don't depend on that number at all, which is why the routing layer is the thing to settle first.

Here's what the routing layer gives teams either way:

- **Regional inference with verifiable residency:** Pin inference and at-rest storage to a US or EU region per request, with no silent cross-region fallback and the resolved region reported on every response.

- **Zero markup on tokens, including BYOK:** Pay the provider's published rate so build-versus-buy comparisons run against a real baseline, with budgets and spend alerts at the team, project, and key scope.

- **Provider and model allowlists:** Enforce an approved provider set team-wide on every request, including bring-your-own-key traffic, with zero data retention and prompt-training controls alongside.

- **Automatic cross-provider failover:** Fallback routing rescued 3.5% of requests and 4.9% of market cost across seven months of production traffic, without per-provider retry logic in application code.

- **One-line model switching and routing rules:** Swap models with a string change, and apply team-wide rewrite or deny rules without touching application code.

[Start a new project](https://vercel.com/new) to put a routing layer in front of your models before committing to hardware, or browse [Vercel templates](https://vercel.com/templates) for AI applications already wired up for multi-provider routing.

## [Copy link to heading](#frequently-asked-questions-about-self-hosted-llms)Frequently asked questions about self-hosted LLMs

### [Copy link to heading](#what-hardware-do-i-need-to-self-host-an-llm)What hardware do I need to self-host an LLM?

A fine-tuned small model runs on a single consumer GPU under 24 GB of memory. Larger open-weight models require a multi-GPU node, and published benchmarks commonly use H100-class hardware, which is rented for roughly $7 per GPU-hour at on-demand cloud list pricing.

### [Copy link to heading](#is-self-hosting-an-llm-hipaa-compliant)Is self-hosting an LLM HIPAA-compliant?

Model location is one component of HIPAA compliance, not the whole of it. Logging, key management, and network egress all have to stay inside the covered environment, and consumer endpoints without a business processing agreement are not compliant, regardless of where the model runs.

### [Copy link to heading](#how-does-a-self-hosted-llm-gateway-differ-from-a-managed-ai-gateway)How does a self-hosted LLM gateway differ from a managed AI gateway?

A self-hosted gateway means running the proxy and its supporting datastores yourself, in exchange for full control of the request path and the option of an air-gapped deployment. A managed gateway operates that layer for you with cross-provider failover, observability, and zero token markup.

### [Copy link to heading](#can-i-use-a-gateway-with-a-self-hosted-llm)Can I use a gateway with a self-hosted LLM?

The routing layer and the model layer are independent choices, so a self-hosted proxy can sit in front of your own inference endpoints alongside managed providers, giving one surface for observability and spend control. AI Gateway routes across its own provider catalog, with bring-your-own-key support for those providers.

### [Copy link to heading](#at-what-volume-does-self-hosting-an-llm-become-cheaper)At what volume does self-hosting an LLM become cheaper?

There's no single threshold, because effective cost depends on your sustained arrival rate, model architecture, and latency target rather than monthly token count alone. Benchmark your own traffic against managed pricing for the same open-weight model rather than against frontier API rates.