Serving an open-weight model starts with one of two commands: ollama run or vllm serve. The published benchmarks struggle to settle which, putting Ollama ahead in some runs and behind in others, sometimes on the same GPU. That disagreement is real, and it resolves once the two engines are read as answers to different questions. Ollama processes one request at a time by default, while vLLM claims most of a GPU at startup so it can fold dozens of requests into a forward pass already in flight. Both produce an OpenAI-compatible endpoint in minutes, though switching later means changing model format, configuration, and the default context window. This guide covers where the two engines split, how each behaves under concurrent load, and what breaks in the migration between them. **Key takeaways:** - Ollama and vLLM optimize for opposite regimes, one developer against one model versus one GPU saturated by concurrent clients. - Published single-user benchmarks disagree with each other because the quantization format and the host CPU move the result more than the engine architecture does. - Under concurrency, the gap is large. On an A100, an Ollama instance tuned for parallelism peaked at 41 tokens per second against vLLM's 793. - Moving from Ollama to vLLM breaks model format, model naming, Modelfile configuration, and the default context window, which silently changes model output. - vLLM's cold start costs development time rather than serving time. Measured startup for a 6.7B model on an H100 fell from 35.77s on v0.8.0 to 16.23s on v0.11.0. ## [Copy link to heading](#ollama-vs-vllm-at-a-glance)Ollama vs vLLM at a glance Request scheduling is where the two engines actually differ. Ollama's OLLAMA\_NUM\_PARALLEL [defaults to 1](https://docs.ollama.com/faq), an accurate statement of what the tool assumes about incoming traffic. vLLM's continuous batching scheduler admits new requests into a forward pass already running, and PagedAttention keeps the key-value (KV) cache dense enough to hold hundreds of sequences at once. That one split accounts for the benchmark spread, the memory behavior, the startup times, and the migration cost. ### [Copy link to heading](#what-this-comparison-is-based-on)What this comparison is based on This comparison draws on the official documentation for both projects and on published benchmarks that name their hardware, model, and quantization format. Concurrency figures come from an instrumented A100 run on server hardware, single-user figures from Blackwell workstations and data center cards. Vendor documentation is authoritative on defaults, formats, and endpoints, and partial by design on anything comparative. Here's how the two split across the dimensions that change a deployment decision, based on the August 2026 releases of each: | Dimension | Ollama | vLLM | | --- | --- | --- | | Core architecture | Single binary wrapping llama.cpp for GGUF, with a separate MLX engine for Apple Silicon | PagedAttention KV cache with continuous batching | | Throughput under concurrency | 41 tok/s peak tuned to 32 parallel requests | 793 tok/s peak on the [same A100 test](https://developers.redhat.com/articles/2025/08/08/ollama-vs-vllm-deep-dive-performance-benchmarking) | | Model format | GGUF | Hugging Face safetensors, with GGUF through an external plugin | | Quantization | GGUF quant types natively, NVFP4 through MLX | AWQ, GPTQ, INT8, INT4, FP8, bitsandbytes, and a quantized KV cache | | Hardware targets | Apple Silicon (Metal and MLX), NVIDIA CUDA, AMD ROCm, Jetson, ARM64 | NVIDIA CUDA, AMD ROCm, Intel XPU, and [Apple Silicon](https://docs.vllm.ai/en/stable/getting_started/installation/) through vLLM-Metal, plus CPU targets | | API surface | OpenAI Chat Completions, Completions, Responses, and Embeddings, plus native /api/\* and Anthropic Messages | OpenAI Chat Completions, Completions, Responses, Embeddings, batch, and audio endpoints | | Memory behavior | Loads on demand, reports its CPU and GPU split per model | Pre-allocates 90% of video RAM (VRAM) at startup by default | | Licensing | MIT | Apache 2.0 | Each row behaves differently on real hardware than the cell suggests. The five that carry the most weight in a real decision are worth walking one at a time. ### [Copy link to heading](#request-scheduling-and-kv-cache-management)Request scheduling and KV cache management Ollama loads a model into memory when a request for it arrives and can keep several loaded at once, with OLLAMA\_MAX\_LOADED\_MODELS defaulting to three per GPU. vLLM runs one model per server process and treats the GPU as a resource to fill, so a request arriving mid-generation joins the current batch instead of queuing behind it. The tradeoff is symmetric. Ollama gives up aggregate throughput to stay flexible about what is loaded, and vLLM gives up flexibility to keep the GPU busy. ### [Copy link to heading](#throughput-under-concurrent-load)Throughput under concurrent load Concurrent load is where the gap gets large enough to change a hardware budget. Across concurrency levels from 1 to 256 on a single A100, vLLM [peaked at 793](https://developers.redhat.com/articles/2025/08/08/ollama-vs-vllm-deep-dive-performance-benchmarking) tokens per second with a P99 time to first token (TTFT) of 80ms. Ollama tuned to OLLAMA\_NUM\_PARALLEL=32 peaked at 41 tokens per second with a P99 TTFT of 673ms. One detail in that run cuts the other way. Ollama's inter-token latency stayed low and stable as concurrency rose while vLLM's climbed, because batching buys aggregate throughput by making individual sequences share a forward pass. ### [Copy link to heading](#model-format-and-quantization)Model format and quantization Ollama serves GGUF with the quantization baked into the model tag, so pulling llama3:8b-q4\_K\_M gets a quantized model with no further configuration. vLLM's primary path is Hugging Face safetensors, with a wider set of backends including AWQ, GPTQ, INT8, INT4, FP8, and bitsandbytes. Its GGUF support [moved out of core](https://docs.vllm.ai/en/stable/features/quantization/gguf/) into a plugin that still carries an experimental warning. The classic mistake is comparing an Ollama Q4 build against a vLLM fp16 build and attributing the gap to the engine, when a large share of it comes from the quantization format on either side. ### [Copy link to heading](#hardware-targets-and-api-surface)Hardware targets and API surface Hardware coverage has converged. vLLM now lists Apple Silicon as a GPU platform through vLLM-Metal alongside NVIDIA CUDA, AMD ROCm, and Intel XPU, while Ollama covers Apple Silicon through Metal and MLX, plus CUDA, ROCm, Jetson, and ARM64. Both also serve OpenAI Chat Completions, Completions, Responses, and Embeddings, so client code moves between them with a base URL change. One vLLM detail deserves attention before anything goes on a network. The --api-key flag only authenticates paths under /v1, /v2, and /inference, leaving /invocations to expose the same inference capability without that check. ### [Copy link to heading](#operational-burden-and-startup-time)Operational burden and startup time Ollama installs as a single binary and allocates memory as models load, reporting the CPU and GPU split for each one. When a model doesn't fit in VRAM, it spreads across available GPUs or falls back partially to system memory, which shows up as reduced throughput rather than an error. vLLM pre-allocates 90% of VRAM at startup by default, which matters before co-locating anything else on the card. Startup also runs torch.compile, captures CUDA graphs, and allocates the KV cache before the first token. That sequence is [predominantly CPU-bound](https://arxiv.org/abs/2606.07362) and falling fast, with a 6.7B model on an H100 starting in 35.77s on v0.8.0 and 16.23s on v0.11.0. Every default in that table says something about the traffic each project's maintainers expect. ## [Copy link to heading](#diving-into-ollama-for-local-development-and-single-node-serving)Diving into Ollama for local development and single-node serving The Ollama server ships as a single binary wrapping llama.cpp for GGUF models, with a separate MLX engine that makes Apple Silicon a first-class target rather than a fallback. Version [v0.33.0](https://github.com/ollama/ollama/releases) shipped in August 2026 under an MIT license. The design goal is visible in the first command a new user runs. ollama run \\ downloads weights and starts generating with no configuration file, no environment variables, and no decisions about memory utilization. ### [Copy link to heading](#pros-and-cons-of-ollama)Pros and cons of Ollama Ollama's case is about time to a working model and about staying out of the way while a team is still deciding what to build. Ollama pros: - **Minutes to first token:** Installation is a single binary and model pulls are one command, with quantization selected by tag rather than by flag. - **Multiple models resident:** Switching models during evaluation doesn't mean restarting a server. - **Graceful memory degradation:** Models that exceed available VRAM spread across GPUs or partially into system memory instead of failing to start. - **Broad client compatibility:** OpenAI-compatible endpoints cover most tooling, and Anthropic Messages support reaches the rest. - **Stable per-user latency:** Inter-token latency stays low and flat as concurrency rises, which matters for interactive single-user work. Ollama cons: - **Serial by default, with a ceiling tuning can't lift:** OLLAMA\_NUM\_PARALLEL is 1 unless changed, raising it multiplies KV cache memory by the parallel count, and throughput plateaus well before vLLM's does. - **A 4,096-token default context:** The default applies regardless of what the model was trained to handle, until OLLAMA\_CONTEXT\_LENGTH or num\_ctx says otherwise. - **GGUF only:** Safetensors checkpoints, including most newly released weights, need a conversion step before Ollama can serve them. - **Silent hardware fallback:** A model that lands partially on CPU still answers requests, so the failure surfaces as slow output rather than a startup error. **Best for:** Ollama fits local development, prototyping, and internal tools where the concurrency ceiling is a handful of people rather than a request queue. It also fits Apple Silicon workstations, where the MLX engine gives it a hardware advantage data center engines were not built to contest. **Pricing:** Ollama is free and open source under MIT. The cost is the hardware it runs on, which for single-node use is a machine the team already owns. ### [Copy link to heading](#where-ollama-stands-out)Where Ollama stands out Ollama pulls ahead on iteration speed, and the mechanism is startup time. Independent measurement of containerized cold starts found llama.cpp reaching a ready state in two to five seconds, against a vLLM startup phase of about 176 seconds for the same 8B model on an A100. For a developer restarting a server after every configuration change, that compounds across a working day in a way no throughput number captures. It's the strongest argument for keeping Ollama in the loop even at teams serving production traffic on something else. Ollama also holds up better than its reputation at batch size 1. A tuned llama.cpp path with no scheduler queue, no paged cache indirection, and no Python layer can beat a data center engine outright on hardware where its quantization format is well supported. That advantage holds only while one request is in flight. ## [Copy link to heading](#exploring-vllm-for-concurrent-production-serving)Exploring vLLM for concurrent production serving vLLM is a Python inference server built around PagedAttention, a block-based KV cache manager, and continuous batching. Version [v0.28.0](https://github.com/vllm-project/vllm/releases) shipped in August 2026 under Apache 2.0. The project originated at UC Berkeley's Sky Computing Lab and is now hosted by the [PyTorch Foundation](https://pytorch.org/blog/pytorch-foundation-welcomes-vllm/). vllm serve \\ starts an OpenAI-compatible server against safetensors weights. Every design decision underneath that command assumes the GPU should be full. ### [Copy link to heading](#pros-and-cons-of-vllm)Pros and cons of vLLM Taking control of memory and scheduling is how vLLM earns its throughput, and it's also where the costs come from. vLLM pros: - **Continuous batching:** Requests join a forward pass already running, so aggregate throughput scales with concurrency instead of flattening. - **PagedAttention KV cache:** Block-based cache management keeps memory dense enough to hold hundreds of concurrent sequences on one card. - **Wide quantization support:** AWQ, GPTQ, INT8, INT4, FP8, bitsandbytes, and quantized KV cache all have supported paths. - **Full trained context by default:** Context length follows the model's configuration rather than a server default. - **Production endpoint coverage:** Chat Completions, Completions, Responses, Embeddings, batch, and audio transcription and translation are all served from one process. vLLM cons: - **Startup measured in tens of seconds:** Weight loading, `torch.compile`, CUDA graph capture, and KV cache allocation all precede the first token. - **90% VRAM pre-allocation:** The default leaves little room for anything else on the card, including a co-located inference server. - **One model per server:** Serving several models means several processes and the memory arithmetic that follows. - **Safetensors-first:** GGUF requires the external vllm-gguf-plugin and remains flagged as experimental. - **Partial API key coverage:** --api-key protects /v1, /v2, and /inference paths, leaving /invocations to be handled at the network layer. **Best for:** vLLM fits teams serving a fixed model to concurrent users on dedicated GPUs, where token throughput per card sets the hardware bill, and batch inference jobs, where startup cost amortizes across a long run. **Pricing:** vLLM is free and open source under Apache 2.0. The real cost of a [self-hosted LLM](https://vercel.com/i/self-hosted-llm) is GPU time plus the engineering attention required for multi-GPU scheduling, memory tuning, and a weekly release cadence. ### [Copy link to heading](#where-vllm-stands-out)Where vLLM stands out The advantage arrives with the second and third concurrent users, and it widens rather than narrows. Tuning Ollama for parallelism narrows nothing, because the bottleneck is scheduling rather than a configured limit. The mechanism matters more than the multiple. Batching converts idle GPU cycles between tokens into work on other sequences, so throughput rises with load until the KV cache fills. Nothing in Ollama's architecture produces that curve, which is why single-instance tuning can't close the gap. Which engine that argues for depends on the traffic a team actually has. ## [Copy link to heading](#choosing-between-ollama-and-vllm-for-your-workload)Choosing between Ollama and vLLM for your workload Regret about this decision usually takes one of two shapes. Some engineering teams ship internal tools on Ollama, watch p99 latency climb as adoption spreads, and discover the ceiling during a demo. Others adopt vLLM during prototyping and spend weeks paying startup costs and memory tuning for concurrency that hasn't arrived. Both are paying for half of the decision they didn't examine. The way to avoid it is to pick a traffic shape, then plan for the switch rather than being surprised by it. ### [Copy link to heading](#choose-ollama-for-iteration-speed-and-single-node-work)Choose Ollama for iteration speed and single-node work For prototyping, local development, and internal tools with a handful of users, Ollama is the stronger call. Startup in seconds instead of minutes is worth more than aggregate throughput when the loop is edit, restart, test. The cost being accepted is a concurrency ceiling well below vLLM's, and a context default that quietly shapes output until someone changes it. ### [Copy link to heading](#choose-vllm-for-concurrent-traffic-on-a-fixed-model)Choose vLLM for concurrent traffic on a fixed model Production serving to concurrent users on dedicated GPUs is where vLLM becomes the stronger call. Continuous batching changes the shape of the throughput curve rather than its height, which nothing else in this comparison does. The same logic applies to batch inference jobs, where a startup cost measured in tens of seconds disappears against a run measured in hours. What gets paid in exchange is operational. VRAM pre-allocation, per-model processes, cold start sequencing, and a weekly release cadence all land on whoever owns the box. ### [Copy link to heading](#choose-against-your-own-hardware,-not-a-published-headline)Choose against your own hardware, not a published headline Batch-size-1 results are dominated by factors outside the engine, which is why published numbers conflict. One set of [Blackwell runs](https://petronellatech.com/blog/ollama-vs-vllm-2026-live-benchmarks-on-blackwell) put Ollama 38 to 68% faster than vLLM on one host, then 40 to 54% slower on a second host with the same GPU and the same models. The variable was the CPU and the platform generation underneath the card. Quantization format moves results the same way. The A100 run that puts vLLM ahead across every concurrency level had Ollama serving fp16 weights rather than the quantized GGUF build a real deployment would use. That tilts the comparison before either scheduler is involved. Any single-user comparison that omits its host CPU and its quantization format for both sides is not measuring what its headline claims. The move from one engine to the other is where the second problem starts, because the two don't swap cleanly. ## [Copy link to heading](#migrating-from-ollama-to-vllm:-what-breaks)Migrating from Ollama to vLLM: what breaks Client code moves with a base URL change. Nothing underneath it does. ### [Copy link to heading](#model-format-and-naming)Model format and naming Ollama commonly distributes GGUF models and can import supported Safetensors checkpoints. vLLM primarily uses Hugging Face checkpoints; its external GGUF plugin remains experimental and under-optimized. Names change with the format. A short tag like llama3.1:8b becomes a Hugging Face repository ID, or whatever string --served-model-name sets, and every hardcoded model name in application code has to follow. ### [Copy link to heading](#context-window-defaults)Context window defaults The default context window in Ollama is 4,096 tokens. vLLM uses the model's fully trained context. If nobody ever set `num_ctx` or `OLLAMA_CONTEXT_LENGTH`, output changes after migrating, frequently getting longer and better, with no code change to explain it. That's the hardest migration failure to diagnose, because nothing errors. ### [Copy link to heading](#modelfile-configuration)Modelfile configuration Modelfiles do not transfer verbatim. Map `FROM` to the vLLM model argument, `PARAMETER num_ctx` to `--max-model-len`, `TEMPLATE` to `--chat-template`, and move `SYSTEM` into application or request configuration. The system prompt is the one that bites. A prompt living in a Modelfile is invisible to application code, so it disappears at migration without anything in the diff to show for it. ### [Copy link to heading](#cold-start-and-the-development-loop)Cold start and the development loop Ollama loads a model in seconds. vLLM loads weights, runs `torch.compile`, captures CUDA graphs for multiple batch shapes, and allocates the KV cache before serving anything, and recent releases have roughly halved that sequence without closing the gap. Containerized deployments make it worse, because a clean-state start pays image pull on top of engine startup. The practical answer is to stop treating the two engines as alternatives during development. The working pattern is to prototype against Ollama, deploy against vLLM, and keep the OpenAI-compatible client code identical across both. ## [Copy link to heading](#how-vercel-helps-teams-running-open-weight-models-beyond-ollama-and-vllm)How Vercel helps teams running open-weight models beyond Ollama and vLLM Both engines assume a team wants to own the serving layer. Self-hosting is the right answer for a real set of workloads. Other teams reach a point where the engine choice has stopped being about model quality and started being about GPU procurement, and the honest move at that point is to stop self-hosting. Two patterns tend to precede that decision, and one set of requirements rules it out. ### [Copy link to heading](#the-serving-layer-starts-consuming-the-engineering-budget)The serving layer starts consuming the engineering budget The pain arrives gradually. Model format conversions, VRAM utilization tuning, cold start sequencing, multi-GPU scheduling, and weekly upstream releases all land on whoever owns the box, and none of it is visible in a roadmap. You notice it when the person who set up the GPU node becomes the person nobody can reassign. The work is real infrastructure work, and it competes directly with the application the models were supposed to serve. Routing open-weight model requests through [AI Gateway](https://vercel.com/docs/ai-gateway) removes that layer entirely. One OpenAI-compatible endpoint reaches hundreds of models across providers, with no markup on tokens, including under [Bring Your Own Key](https://vercel.com/docs/ai-gateway/authentication-and-byok/byok). Open-weight models accounted for [29% of gateway volume](/blog/ai-gateway-production-index-july-2026) in July 2026, so this is a well-worn path rather than an edge case. ### [Copy link to heading](#provider-and-hardware-failure-handling-lands-in-application-code)Provider and hardware failure handling lands in application code Self-hosting concentrates failure in one place. A node that runs out of VRAM, a driver update that breaks a kernel, or a model that no longer fits after a quantization change all take the endpoint down. The retry logic then ends up written into the application by whoever was on call. AI Gateway moves that to the routing layer, where [model fallbacks](https://vercel.com/docs/ai-gateway/models-and-providers/model-fallbacks) reroute around a failing provider without a deploy. Adding a proxy hop usually costs latency, and here it doesn't. Requests route across 126 points of presence and stay largely on managed network rather than the open internet. A week-long production A/B test against a previous router improved [P99 streaming latency](/blog/cline-on-ai-gateway) by 10 to 14% across the most-used models. ### [Copy link to heading](#some-requirements-rule-a-managed-gateway-out)Some requirements rule a managed gateway out A managed gateway gives up control over hardware selection and model configuration, which rules it out for a real set of workloads. Custom quantization, fine-tuned model serving, strict data residency requirements, and air-gapped environments all need an engine a team runs itself. Sustained high token volume on a single model can also favor owning the GPU, because the cost curve of a dedicated card at full utilization is different from per-token pricing. Anyone in those categories should keep Ollama or vLLM and treat the engine choice as the real decision, not a stepping stone. Outside them, the serving layer is infrastructure work that a routing layer would absorb. The tradeoffs between self-hosting and a managed layer get more room in the guide to [open source AI gateways](https://vercel.com/i/open-source-ai-gateways). ## [Copy link to heading](#ship-open-weight-models-without-owning-the-serving-layer)Ship open-weight models without owning the serving layer The published benchmarks stop contradicting each other once the question becomes which traffic regime a project is in, and that regime changes between prototype and production. Prototyping rewards startup speed, production rewards throughput per GPU, and the transition costs a real migration rather than a configuration change. The engine choice is worth getting right, and it's worth deciding separately whether the serving layer belongs to the team at all. Vercel handles the layer underneath either answer: - **AI Gateway:** One OpenAI-compatible endpoint reaches hundreds of models, open-weight included, with zero markup on tokens and the same terms under Bring Your Own Key. - **Model fallbacks:** Requests reroute around a failing provider without a deploy, which removes the retry logic that otherwise accumulates in application code. - **AI SDK:** Model selection is [a single string](https://vercel.com/docs/ai-sdk), so moving between a self-hosted endpoint and a managed one stops being a rewrite. - **Fluid compute:** Model calls run on [Fluid compute](https://vercel.com/docs/fluid-compute), where concurrent request handling cuts the cost of I/O-bound work that spends most of its wall clock time waiting on inference. - **Gateway observability:** Per-request token counts, latency, and spend are [logged in-path](https://vercel.com/docs/ai-gateway/observability-and-spend/observability), so cost tracking needs no instrumentation in application code. [Start a Vercel project](https://vercel.com/new) and ship your first model call on the first git push, or browse [vercel.com/templates](https://vercel.com/templates) to begin from a working AI application and swap in the open-weight models you want. ## [Copy link to heading](#frequently-asked-questions-about-ollama-vs-vllm)Frequently asked questions about Ollama vs vLLM ### [Copy link to heading](#can-ollama-serve-production-traffic)Can Ollama serve production traffic? Yes, for small teams, and through horizontal scaling rather than single-instance tuning. A tuned instance plateaued at 41 tokens per second on an A100, so the durable pattern is multiple Ollama instances behind a load balancer. ### [Copy link to heading](#do-ollama-and-vllm-both-expose-openai-compatible-apis)Do Ollama and vLLM both expose OpenAI-compatible APIs? Both serve Chat Completions, Completions, Responses, and Embeddings, so OpenAI SDK code moves between them with a base URL and model name change. Ollama adds Anthropic Messages compatibility and native /api/\* endpoints. vLLM adds batch and audio endpoints. ### [Copy link to heading](#which-engine-is-faster-for-a-single-user)Which engine is faster for a single user? It depends on quantization and the host CPU more than on the engine. Published single-stream results put Ollama's quantized builds anywhere from 68% ahead of vLLM to 54% behind it, on the same GPU, with the platform underneath the card explaining most of the spread. ### [Copy link to heading](#how-do-you-serve-open-weight-models-without-running-an-inference-server)How do you serve open-weight models without running an inference server? A managed gateway handles it, trading hardware control for zero operational surface. The application code is the same OpenAI-compatible client either way, so the switch is a base URL and a model string rather than a rewrite. Custom quantization, fine-tuned serving, and data residency still require self-hosting.