When it comes to production traffic across enough model providers, every request is a bet on cost, latency, and quality at once. Some prompts are wasted on a frontier model. Others fall apart on a cheap one.
LLM routing is the layer that reads each request and sends it to the model best suited to handle it, weighing cost, capability, latency, and resilience. Cost-based routing is where almost every team starts, so that’s where we’ll start too, before the strategies that earn their complexity.
Key takeaways:
Cost-based routing uses static per-token pricing as its sole signal and cannot account for queuing delays, degraded quality, or runtime load.
A nominally cheap model under heavy load can cost more per useful response than a pricier model with spare capacity.
Fallback chains are the cheapest insurance against provider outages and keep retries, cross-provider fallbacks, and circuit breakers as three distinct mechanisms.
Text-to-SQL benchmarks show routing savings depend entirely on how aggressively the router shifts traffic off the highest-cost model.
Routing is a layered control system rather than a single trick and belongs behind a control plane measured against production, not price sheets.
Copy link to heading6 LLM routing strategies at a glance
When I sketch routing for a new system, I start by putting the strategies side by side, because the cheapest one to build is rarely the one the workload needs. Routing signals run from the static token price all the way to the query’s meaning, with fallback chains sitting underneath it to ensure uptime.
Most production systems I’ve run end up combining two or more, layering cost control, quality matching, and resilience rather than committing to a single approach.
Each strategy differs in its primary signal, the latency it adds, the quality control it provides, and the cost of implementation:
The strategies near the top of the table are the ones you can stand up in an afternoon. The ones lower down give you more control over cost and quality, but they ask for real instrumentation in return. I’ll build them up in that order, starting with the one almost everyone reaches for first.
Copy link to heading1. Cost-based routing
Cost-based routing is the first thing every team I know reaches for, and for a defensible reason. It sends each request to the cheapest model deployment that can handle it, using static per-token pricing as the only signal and performing no analysis of the query itself.
When model prices differ by an order of magnitude, that spread alone can make routing financially material, so cheaper tasks run on economical models and the demanding ones get the capable deployments.
Copy link to headingThe limitation that matters
The trap I’ve watched teams fall into is treating the price sheet as ground truth. Static pricing doesn’t reflect what happens at runtime, and a nominally cheap model under heavy load can deliver a higher cost per useful response than a pricier model with capacity to spare.
Cost-based routing has no way to see queuing delays, degraded quality, or runtime load. That blind spot is why I never let a system stop at price alone. Pairing cost with a runtime signal is the obvious next move, and latency is the easiest one to add.
Copy link to heading2. Latency-based routing
The moment response time becomes visible to a user, I care less about list price and start caring about which deployment answers fastest. Latency-based routing directs each request to the deployment with the lowest observed response time, tracking response times for each deployment and sending new requests to the quickest one at that moment.
In practice, I expose latency as a sortable parameter, set a preferred maximum threshold, and let providers that exceed it be deprioritized while remaining available as fallbacks.
Copy link to headingWhy historical latency is a lagging indicator
What burned me early was trusting a latency number that was already stale. Observed latency is a trailing signal. It blends queue wait time with per-token processing speed, so a deployment that looked fast a second ago can bog down the instant traffic shifts toward it.
That makes latency-based routing a strong fit for search, support chat, and consumer assistants, where users feel every extra second, and a weak indicator on its own of whether a given model is right for a given request.
Matching the request to the model is the gap that semantic routing exists to close.
Copy link to heading3. Semantic routing
Semantic routing is when routing stops being plumbing and becomes a product decision. Instead of looking at price or speed, it reads the content and intent of each query and selects the model with the strongest demonstrated capability for that task.
The router is deciding based on what the user is asking and which task type the request represents, which is a different and harder question than any of the signals above it.
Copy link to headingHow semantic matching works
When I’ve built these, two implementations keep showing up:
Embedding-based routing converts the query into a vector and measures similarity to labelled reference prompts, each representing a task category.
Classifier-based routing uses a lightweight model, a BERT variant, an MLP, or a fine-tuned small LLM, trained on task labels to push coding work to code-optimized models, summarization to small fast ones, and reasoning to higher-capability deployments.
The payoff is real because a well-matched smaller model routinely beats a larger generalist on a task the generalist was never tuned for.
Copy link to headingWhere semantic routing needs support
No classifier I’ve shipped has ever caught every edge case. Semantic routing works best when the router learns which model wins for each request type and then hands off to a recovery path the moment the classification is wrong or a provider drops out.
That recovery path is almost always a fallback chain, which is the next layer to build.
Copy link to heading4. Fallback chains
If I could keep only one routing strategy in production, it would be this one. Fallback chains define an ordered sequence of model deployments, attempting the primary one first and cascading to the next only when a specific condition is triggered, such as an error, a context-size mismatch, or a provider going dark.
It is the cheapest insurance you can buy against the reality that every provider has bad days.
Copy link to headingThe three-layer resilience stack
The mistake I see most is collapsing three different mechanisms into one.
Retries, fallbacks, and circuit breakers solve distinct failure modes and run in a fixed order:
Retries first inside a model group
Fallbacks across providers
Circuit breakers gating whether either is even attempted
Vercel’s AI Gateway implements the fallback layer declaratively through a models array, including via providerOptions.gateway.models, where requests route to backup models in order when the primary is unavailable.
Keeping the three mechanisms separate is what keeps recovery logic out of scattered application code.
Copy link to headingCoordinating retries across layers
Fallback chains only behave if you coordinate retries across the whole stack. If your routing layer waits for retries to exhaust before switching providers, it stacks its own retry attempts on top of those already baked into your SDK and agent loop.
In an agentic system, that multiplication is how a minor blip becomes a self-inflicted traffic surge, a failure mode I’ll come back to. For now, the point is that fallback chains pair naturally with the next strategy, where the targets are interchangeable rather than ranked.
Copy link to heading5. Weighted load balancing
Load balancing is the strategy I reach for when all my targets are genuinely interchangeable. It spreads traffic across equivalent deployments using configured weights, rate-limit awareness, or least-busy selection.
This way, it treats every target as a substitute for the others and optimizes for throughput and availability rather than for matching a request to a particular model’s strengths.
Copy link to headingTraffic distribution patterns
The patterns I lean on come down to three:
A weighted random distribution follows the configured RPM or TPM weights.
Rate-limit-aware routing tracks the remaining capacity per endpoint and steers away from deployments near their capacity limits.
Least-busy routing sends each request to the deployment with the fewest in-flight requests.
These sit comfortably alongside fallback chains, where a load-balanced cluster absorbs normal traffic, and only a full cluster failure trips the cross-provider backup.
That split keeps the steady-state distribution simple while preserving a clean recovery path. It is the right call when the goal is to use capacity well rather than to match requests to different model strengths.
Copy link to headingSession affinity for multi-turn conversations
Multi-turn conversations are where stateless routing quietly breaks, and I’ve had to retrofit session affinity more than once. Sticky routing maps a session identifier to a specific deployment, preserving conversation context and ensuring consistent behavior for A/B testing.
It also addresses multi-turn routing limitations that surface when a stateless router evaluates every turn in isolation. Load balancing distributes traffic evenly across healthy providers, and session affinity preserves continuity for workloads that need it.
Copy link to heading6. Budget-ceiling routing
Budget-ceiling routing is the one I wish more teams reached for before turning to a spreadsheet of token prices. It enforces a hard or soft cost ceiling per request or per workflow, then picks the highest-capability model that fits inside that ceiling.
The difference from pure cost-based routing is the whole point: cost-based routing always selects the cheapest option regardless of capability, whereas this approach combines cost awareness with quality optimization.
Copy link to headingConstraint-aware model selection
I think of this as a constrained optimization problem rather than a price lookup. The router balances response quality against cost and latency limits using more than a static pricing table. In practice, it comes down to two policies.
One picks the lowest-price model capable of handling the query. The other matches model tier to query complexity, then downgrades later steps when the running total threatens the budget. The important shift is that the router no longer treats every step in a workflow as equally valuable.
This lets you reserve the expensive models for the steps where capability moves the outcome.
Copy link to headingThe calibration warning
Here’s the warning I give every team before they promise anyone a number. Benchmarks in text-to-SQL are a useful counterpoint to routing optimism, because savings depend entirely on how aggressively the router shifts traffic off the highest-cost model.
Calibration determines the result. Tune the router to avoid any quality regression, and it keeps routing most queries to the expensive model, so the savings evaporate. Measure the actual fraction of production traffic that lands on cheaper models before you project anything based on list-price differences.
Even a perfectly calibrated router only handles the normal case, which is never where the trouble starts.
Copy link to headingHow to build routing into your deployment workflow
Every pattern I’ve described needs the same thing underneath it: a control plane that sits between your application code and the model providers and owns provider selection, fallback logic, structured output enforcement, and observability.
Vercel places that control plane at the platform layer, so you adopt these patterns via configuration rather than bespoke code.
The AI SDK routes to supported providers via a single model parameter, such as anthropic/claude-sonnet-4, so switching providers or adding a fallback never touches application logic.
AI Gateway takes a models array, where requests cascade to backups in case of errors, context-size mismatches, or provider unavailability. providerOptions.gateway controls which providers are tried, in what order, and which are excluded. This is where I encode compliance and cost preferences.
The same layer is also where consistency and visibility live, since the AI SDK’s structured output curbs format drift and its usage reporting, latency tracking, and OpenTelemetry integration give you the data to judge whether your routing decisions are working.
Copy link to headingPut your routing behind a control plane
If you take one thing from how I run multi-model traffic, make it this. Routing is not a single trick; it’s a layered control system. Cost-based routing is a fine place to start; latency adds runtime awareness; semantic routing fixes task matching; and fallback chains, plus load balancing, keep traffic moving when a provider or deployment changes state beneath you.
The implementation always depends on your traffic shape, your model mix, and the level of reliability you need, but the operating principle holds. Put routing behind a control plane and measure what happens in production rather than what the price sheets promise.
Vercel’s primitives are built to make that control plane something you configure rather than maintain:
AI Gateway: Declarative fallback chains through a
modelsarray, with provider order, inclusion, and exclusion controlled per requestAI SDK: A single model parameter that swaps providers and adds fallbacks without application code changes
Structured output:
Output.object()andOutput.array()enforcement that normalizes JSON shape across providersBuilt-in observability: Usage reporting, latency tracking, and OpenTelemetry integration for judging routing decisions in production
Provider controls:
providerOptions.gatewaysettings that encode compliance constraints and cost preferences at the routing layer
Create a new Vercel project or explore existing templates to put a routing control plane in front of your model traffic today.
Copy link to headingFrequently asked questions about LLM routing
Copy link to headingWhat is the difference between LLM routing and load balancing?
Load balancing distributes traffic across identical instances of the same model, while LLM routing selects which model to use for a request based on query complexity, cost, or capability. In a typical setup, a gateway routes to the right model first, then load-balances across that model’s instances.
Copy link to headingHow much latency does a router add to each request?
Rule-based and cost-based routing add minimal overhead, since they decide on static signals. Semantic routing costs more because analyzing a query’s content and intent before selecting a model is real work that simpler strategies skip entirely.
Copy link to headingCan LLM routing improve accuracy and reduce cost?
Yes, and the two often move together. Studies and production deployments both show that routing improves accuracy by sending requests to models better suited to the task than a larger generalist would be, while keeping cheaper models busy on simpler work.
Copy link to headingWhen should I use routing versus fine-tuning a single model?
Fine-tuning suits cheaper inference on a narrow task when you have enough training data, while routing dispatches different query types to specialized models at inference time. They are complementary, and fine-tuned models are common routing targets rather than an alternative to routing.