A mixture of experts model can store far more parameters than it uses for any single token, and that gap is the entire point of the architecture. Instead of running every parameter on every input the way a dense model does, it routes each token to a small subset of specialized subnetworks, so a model's total size and its per-token compute cost stop being the same number.
Copy link to headingKey takeaways
A mixture of experts model replaces the single feed-forward network in each transformer block with parallel expert subnetworks, and a learned router activates only a few of them per token.
Sparse activation separates capacity from compute. DeepSeek-V3 stores 671B parameters but activates 37B per token, so its forward-pass compute is closer to that of a 37B dense model than a 671B one.
Memory footprint tracks the total number of parameters because routing decisions occur at inference time, so every expert must reside in GPU memory whether or not it activates for a given token.
Routers can concentrate traffic on a small set of favored experts during training, leaving the rest undertrained, so model developers apply load-balancing techniques before release.
Total parameter count is a poor predictor of API cost or latency for mixture-of-experts models, since billing is per-token and performance depends on how each provider serves the experts.
Copy link to headingWhat is a mixture of experts model?
A mixture of experts model is a neural network in which a learned router activates a small subset of specialized subnetworks, called experts, for each input token. In a mixture-of-experts transformer, the single feed-forward network in each transformer block is replaced with multiple parallel feed-forward experts. The attention mechanism stays unchanged, since only the feed-forward layer gets this treatment.
A dense model works differently: every parameter processes every token, so capacity and per-token compute scale together. Mixture of experts breaks that link through sparse activation: the practice of running only a subset of a model's parameters on a given input, rather than all of them. Total capacity, meaning all the knowledge stored across every expert, can grow by adding experts, while per-token compute stays pinned to the few experts that actually run.
Copy link to headingDense versus sparse activation
A dense model activates every parameter for every token. Each token passes through the same full feed-forward network, so the compute cost of a forward pass is fixed by the model's total size. A larger dense model always costs more to run per token than a smaller one, with no way to add capacity without adding proportional compute.
A sparse model activates only a fraction of its parameters per token. The feed-forward layer is split into several parallel experts, and a router selects a small subset to run for each token instead of running all of them. Qwen3-235B-A22B, a documented Alibaba model, illustrates the gap this creates: it has 128 experts per layer and routes each token to 8 of them. The model has 235B total parameters but uses only 22B per token, so its forward-pass compute is closer to that of a 22B dense model than its total parameter count suggests.
Roughly 9% of the model works on any given token. The other 91% sits idle for that token but stays available for the next one, which may need a different combination of experts entirely. That's the structural difference between the two: a dense model's compute is fixed by what it stores, while a sparse model's compute is fixed by what it selects, and something has to do that selecting for every token that arrives.
Copy link to headingHow mixture of experts routing works
Sparse activation only works because something has to decide, for every token, which few experts get to run. That decision happens in a fixed sequence at every layer:
Scoring: The router, sometimes called the gating network, trains alongside the rest of the model and produces a score for every expert given the current token.
Selection: The router keeps only the
top-khighest-scoring experts, where k is the number of experts allowed to run. Qwen3 MoE variants activate 8 of 128, Kimi K2 activates 8 of 384 plus a shared expert, and DeepSeek-V3 activates 8 of 256 routed experts plus one shared expert that runs on every token.Execution: Only the selected experts process the token. Every other expert in that layer does no work for that token, which is the compute-saving basis on which sparse activation is built.
Mixing: The router's scores double as mixing weights. If one expert scored 0.7 and another 0.3, the final output is a 70/30 blend of their two outputs, replacing what a single dense feed-forward network would have produced.
This sequence repeats independently at every layer, so the same token can hit a different combination of experts as it moves through the model, and expert specialization emerges during training without any subject being assigned. An API caller sees only the final blended output, never the routing decisions behind it. The same instinct, computing only where it's needed, shows up one layer up the stack in retrieval frameworks like LlamaIndex, where a router sends a query to the right index or tool rather than searching everything, though that routing occurs between application components rather than between subnetworks in a single forward pass.
That routing decision also breaks the link between a model's size and its runtime cost. Once only a handful of experts compute per token, total parameter count stops describing the compute a request actually uses, which is exactly the gap the next section explains.
Copy link to headingWhy mixture of experts matters for cost-efficient frontier models
Sparse activation matters because it changes what determines a model's cost, and that shift comes down to three distinct reasons:
Capacity and compute stop scaling together. In a dense model, adding parameters always means adding compute per token. Mixture of experts lets a model store more knowledge without a proportional rise in per-token cost, since only the selected experts run.
Active parameters set the price of a request. Because compute scales with the active count, a request to a trillion-parameter mixture-of-experts model can cost about the same as a request to a much smaller dense model, provided the active fraction is similarly small.
The active fraction is a design choice: It varies across open-weight models. A lower fraction pushes cost down further but concentrates more of the model's knowledge into fewer experts per step, which is the tradeoff the next section covers.
Mixture of experts is the architecture that lets total capacity grow faster than the compute bill. The same routing mechanism that produces those savings is also where the architecture's costs show up.
Copy link to headingTradeoffs and solutions for mixture of experts models
Routing is what makes sparse activation efficient, and it is also where the structural costs arise. Training has to keep expert usage balanced, serving has to keep every expert available before the router knows which ones a token needs, and splitting experts across hardware adds a communication cost of its own.
Copy link to headingKeep routing stable through load balancing
A router can favor a small set of experts as training progresses, since experts that perform well early receive more tokens, train faster, and become even more likely to receive future tokens. Left uncorrected, this produces uneven expert usage: a smaller working set handles most tokens, while the rest receive less training signal, and quality becomes inconsistent across inputs that should be routed similarly.
Model developers manage this before release by applying load-balancing pressure that spreads tokens more evenly across experts. DeepSeek-V3 takes an auxiliary-loss-free approach, adjusting routing scores directly when experts become imbalanced rather than adding a balancing penalty to the training objective. Either approach can be applied during training, so a team calling the finished model via an API inherits the result without managing routing itself.
Copy link to headingManage memory footprint with quantization and offloading
Every expert must reside in GPU memory, even though only a few are active per token, because the routing decision is made at inference time. The router can't know which experts a token needs until it sees the token, so all expert weights must be loaded and ready before the forward pass runs. Mixtral 8x7B computes like a roughly 13B dense model but occupies memory like a 47B one, over 90 GB at FP16, the 16-bit floating-point format commonly used for model weights.
Quantization, which stores weights in lower-precision formats, reduces the memory required by a smaller mixture of experts model and can bring it within reach of a single GPU. Expert offloading keeps rarely used experts in system memory instead of GPU memory, trading some latency for a smaller footprint. With total parameter footprints in the hundreds of billions, frontier-scale models can still require multi-GPU serving even when these techniques are applied.
Copy link to headingReduce communication overhead with expert placement
Serving a mixture-of-experts model across multiple GPUs means every forward pass must send tokens to the device that holds its selected experts, then gather the results. Microsoft's DeepSpeed-MoE research documents this as an all-to-all communication step whose latency scales with the number of devices involved, and it does not automatically overlap with computation the way other distributed-training communication does.
The fix is placement and scheduling rather than routing itself: keeping frequently paired experts on devices with fast interconnects, batching enough tokens per step to keep every loaded expert busy, and using communication libraries tuned to this specific all-to-all pattern rather than generic collectives. None of this is visible to a team calling the model through an API, but it's why identical model weights can post different throughput and latency depending on how a provider has placed them.
Copy link to headingWorking with mixture of experts on the Vercel AI Gateway
AI Gateway is a single endpoint for calling models across AI providers, with unified billing, automatic failover, and usage tracking in a single dashboard, rather than a separate integration per provider. Many of the models available through it are mixture-of-experts architectures, including Kimi K2, DeepSeek V3, and Qwen3-235B-A22B. The three practices below apply across all of them.
Copy link to heading1. Consider cost implications when routing between MoE and dense models
Active parameters give teams a better starting point for MoE cost and latency than headline size, since billing follows the parameters a request actually uses rather than the total a model stores. A trillion-parameter MoE model can cost about the same to call as a much smaller dense model if its active fraction is similarly small, which is the opposite of what the headline parameter count would suggest. The AI Gateway pricing page has current billing details for any model before you commit traffic to it.
Copy link to heading2. Account for provider-level differences in serving the same model
Identical MoE weights can still perform differently across providers, since the serving stack has to place experts across GPUs and coordinate the all-to-all communication described above. AI Gateway's provider management means that serving choice can change without touching application code. When you have a provider preference, pass an ordered list of provider options through providerOptions.gateway.order. For example, ordering Kimi K2.6 across its current providers looks like this:
import { streamText } from 'ai';
const result = streamText({ model: 'moonshotai/kimi-k2.6', prompt: 'Summarize the tradeoffs of sparse activation', providerOptions: { gateway: { order: ['fireworks', 'togetherai'], }, },});
That configuration tries Fireworks first and falls back to Together AI, without changing the rest of the request.
Copy link to heading3. Set up failover across MoE and dense models
Provider diversity keeps model access flexible as traffic changes, but a full model outage needs a different fallback than a slow provider. Model fallbacks let you configure a models array that the Gateway tries in order, and the mechanism is architecture-agnostic, since a MoE model can fall back to a dense one, or vice versa, because the Gateway routes at the model and provider levels.
Copy link to headingReason about active parameters, not headline size
A larger stored model whose per-token compute tracks a smaller active subset resolves once you separate what a model stores from what it activates. For a developer choosing models through an API, active parameter behavior and provider serving quality tell you more about real cost and latency than total size ever will, and a mixture-of-experts architecture is the one that makes those two numbers diverge.
Vercel is the platform where that choice turns into a running application. It hosts the app and, through the AI Gateway and AI SDK, gives a team a single endpoint for calling MoE and dense models side by side, with provider routing, failover, and cost tracking handled under the hood rather than built by hand.
Here's how that comes together in practice:
Unified routing: AI Gateway calls MoE and dense models via the same endpoint, so switching between them is a model-string change rather than a new integration.
Cost visibility: Usage and spend are tracked per model and provider, so active-parameter economics show up on the dashboard rather than as a surprise on the bill.
Automatic failover: Model fallbacks and provider ordering keep a request moving to the next option when one model or provider is unavailable.
No infrastructure to build first: Teams get this routing layer without having to stand up their own provider integrations or load-balancing logic.
Model choice stays task-driven: Teams still pick models based on quality and latency for the task at hand, within cost constraints, rather than by total parameter count.
Start a new project at vercel.com/new or begin from a working example at vercel.com/templates to start routing between MoE and dense models today.
Copy link to headingFrequently asked questions about mixture of experts
Copy link to headingIs a mixture of experts model the same as an ensemble of models?
No. A mixture-of-experts model differs from an ensemble because its experts are internal subnetworks trained jointly with a shared router, and only a sparse subset is activated per token. An ensemble combines multiple model outputs externally, adding coordination outside the model rather than routing it inside.
Copy link to headingDo mixture of experts models need more GPU memory than dense models?
Yes, relative to their active parameter count. All experts must be resident in memory because routing happens at inference time. Mixtral 8x7B, for example, computes like a 13B dense model but needs the memory footprint of a 47B one.
Copy link to headingCan a mixture of experts model run on a single GPU?
Smaller ones can be achieved with quantization, which stores weights in lower-precision formats to reduce memory use. Frontier-scale mixture-of-experts models generally require multi-GPU clusters, and expert offloading can further reduce memory needs while introducing a latency trade-off.
Copy link to headingWhy do some AI providers not disclose whether their models use a mixture of experts architecture?
Because architecture disclosure is partial and model-specific. Some model pages publish size and routing details, including active-parameter counts, while others don't include the information a team would need to classify the model. When architecture isn't disclosed, teams can compare models by task quality, latency, and cost within their own production workflows.