Mixture of Experts

What Is Mixture of Experts?

Mixture of experts (MoE) is a neural network architecture in which each token activates only a small subset of the model's parameters, chosen by a learned router. The model can hold a very large total parameter count while spending the compute of a much smaller one on every request, which lowers inference cost.

Key Takeaways

• MoE separates model size from compute cost: a model can have hundreds of billions of total parameters while activating only a fraction per token.
• The "experts" are interchangeable sub-networks selected by a router at every layer, not specialists in human topics like law or medicine.
• Most frontier and open-weight flagship models now use MoE because it delivers better quality per dollar of inference than dense models at the same cost.
• For self-hosting, MoE shifts the bottleneck to memory: all experts must be loaded even though only a few run per token.

How It Works

A standard dense transformer model runs every input through every parameter. MoE replaces the feed-forward block in each transformer layer with a pool of parallel sub-networks called experts, plus a small router network. For each token at each layer, the router scores all experts and sends the token through the top few, commonly two out of anywhere from eight to well over a hundred. The outputs of the chosen experts are combined and passed on. Since only the selected experts compute, the number of active parameters per token stays small even as total parameters grow. Mistral's Mixtral 8x7B shows the ratio in practice: each token has access to 47B parameters but only 13B are active during inference, and the 2024 paper reports it matching or beating Llama 2 70B and GPT-3.5 on every benchmark evaluated [1].

Training keeps the router honest with load-balancing objectives, because a lazy router that funnels everything to a few favorite experts wastes the rest of the network. The idea also pays off during training itself: Google's 2021 Switch Transformer used sparse routing to pre-train models of up to a trillion parameters, reaching up to 7x faster pre-training than T5 baselines on the same computational budget [2]. At inference time the economics are distinctive. Compute per token tracks active parameters, so latency and per-token cost resemble a smaller model. Memory does not get the same discount: every expert has to sit in accelerator memory ready to be chosen, so serving an MoE takes the VRAM of the full parameter count. That is why API providers love the architecture, since they amortize memory across many users, while a single developer running one model locally feels the memory cost directly. Combined with an inference approach like quantization, MoE is a large part of why capable model access got cheap.

Example

A platform team runs an internal code-review agent that processes every pull request across the company, several thousand a day. On a dense frontier model the per-review token cost made full rollout hard to justify, so they switched the routine tier to an open-weight MoE model with a large total parameter count but only a small fraction active per token, hosted through an inference provider. Review quality on style, bug-spotting, and test-coverage comments stayed within their eval tolerance, while per-review cost dropped far enough to run on every PR instead of a sampled subset. High-risk diffs still escalate to a stronger reasoning model. Nothing about the agent changed; the economics of the model underneath did.

What People Get Wrong

People hear "experts" and picture a committee of domain specialists, one for Python, one for poetry, with the router acting as a dispatcher you could steer. Real experts do not map to human categories. Routing happens per token, per layer, and the specializations that emerge are statistical patterns, often syntactic or positional, that rarely align with topics. You cannot prompt your way to a particular expert, and no expert can be pulled out as a standalone small model. The routing is an internal efficiency mechanism, invisible at the API level.

FAQ

Why do so many LLMs use mixture of experts now? Because it improves the quality-to-cost ratio. Scaling total parameters raises capability, but dense scaling raises inference cost in step. MoE lets labs grow capacity while holding per-token compute roughly flat, which shows up for buyers as stronger models at lower API prices. DeepSeek-V3 is the pattern at frontier scale: 671B total parameters with only 37B activated per token, trained in 2.788 million H800 GPU hours according to its 2024 technical report [3].

Does mixture of experts change how I prompt or build agents? No. Routing is invisible at the API. Where it matters is procurement and hosting: MoE models often undercut dense competitors on price per token, and if you self-host, plan memory around total parameters, not active ones.

Is a mixture of experts model less consistent than a dense model? Routing decisions can make outputs a bit more sensitive to small input changes, and batching effects mean exact reproducibility is not guaranteed even at temperature zero. In practice this is minor; dense models served at scale show similar nondeterminism.

Sources

1. Mistral AI (arXiv). "Mixtral 8x7B uses 13B active of 47B total parameters per token and matches or beats Llama 2 70B and GPT-3.5 across benchmarks." https://arxiv.org/abs/2401.04088. Accessed August 2026.
2. Google Research (arXiv). "Switch Transformer pre-trains trillion-parameter sparse models with up to 7x faster pre-training than T5 at the same compute." https://arxiv.org/abs/2101.03961. Accessed August 2026.
3. DeepSeek-AI (arXiv). "DeepSeek-V3 activates 37B of 671B parameters per token and was trained in 2.788M H800 GPU hours." https://arxiv.org/abs/2412.19437. Accessed August 2026.