Mixture of experts

How it works

A mixture of experts (MoE) layer has two parts: a set of experts and a router. In a language model, the MoE layer replaces the feedforward block of a transformer.

For each token, the layer does three steps:

1. The router gives a score to each expert. In Mixtral, the score is a softmax of the top logits of one linear layer.
2. The router keeps the top experts and ignores the others. Mixtral keeps 2 of 8. Switch Transformer keeps 1.
3. The selected experts operate on the token. The layer adds their outputs with the router weights.

Jacobs, Jordan, Nowlan and Hinton first described the method in 1991. Each of their “many separate networks” learns “a subset of the complete set of training cases”. In 2017, Shazeer and colleagues made it sparse, with thousands of experts and 137 billion parameters at most.

Why it matters

MoE divides two numbers that are the same in a dense model: total parameters and active parameters.

Model	Experts	Total parameters	Active for each token	Source, date
Mixtral 8x7B	8 per layer, 2 selected	47B	13B	Mistral AI paper, Jan 2024
DeepSeek-V3	not given here	671B	37B	DeepSeek-AI report, Dec 2024
Llama 4 Scout	16	109B	17B	Meta blog, Apr 2025
Llama 4 Maverick	128 routed, 1 shared	400B	17B	Meta blog, Apr 2025

The compute for each token follows the active number. Switch Transformer reported “up to 7x increases in pre-training speed” with the same computational resources.

In practice

Three problems come back in each MoE paper:

• Load imbalance. Shazeer and colleagues found that the router “tends to converge to a state where it always produces large weights for the same few experts”. Switch Transformer adds one more loss that pushes for equal routing. DeepSeek-V3 uses a balance method with no added loss.
• Full experts. An expert has a fixed capacity in many systems. In Switch Transformer, a token that finds its expert full skips the expert and goes directly to the next layer.
• Hard training. The Switch authors name “complexity, communication costs and training instability” as the barriers to wide use.

Many open-weight models use MoE: Mixtral (Apache 2.0), DeepSeek-V3 and Llama 4. For local use, the memory of all experts is the limit. Quantization makes the weights smaller.

Security risks of MoE routing

The router makes a new attack surface for a shared inference service. Two research papers show it:

1. Buffer overflow (Hayes, Shumailov and Yona, Feb 2024). Some routing methods use the other items in a batch. Then malicious queries can change the output for benign queries in that batch. The attack needs an expert capacity limit. The authors show it only in “a toy experimental setting”.
2. Prompt theft (Yona, Shumailov, Hayes and Carlini, Oct 2024). An attacker in the batch of a victim can get the full prompt of the victim through Expert Choice Routing. The test used two layers of Mixtral and about 100 queries for each token. The authors say the attack is “infeasible at present” against real world systems. It is a form of LLM data leakage.

The authors give these controls for the operator of the service:

• Keep the inputs of different users isolated in a batch.
• Put the inputs of a batch in a random sequence.
• Give the experts more capacity, to stop token overflow.
• Add randomness to expert selection. For example, sample experts from the router weights and do not use the top k.

Questions and answers

Is a mixture of experts model cheaper to run than a dense model?

A mixture of experts model uses less compute for each token than a dense model with the same total parameters. Only some experts operate on each token. The model does not use less memory. For Mixtral 8x7B, the authors relate the memory cost to all 47B parameters. Each token uses only 13B.

Does each expert know one subject, such as code or biology?

Not in the tests that the Mixtral authors published. They "do not observe obvious patterns in the assignment of experts based on the topic". They found some patterns of syntax and position. The router learns its selections in training. The word "expert" is a name, not a promise of a specialty.

Do the MoE routing attacks work against real services?

Not as published. The buffer overflow attack of Hayes and colleagues was a proof of concept in a toy setup. The prompt theft attack of Yona and colleagues used two layers of Mixtral. Its authors say that it is "infeasible at present" against real world systems. The two attacks work only if the attacker and the victim are in one batch.