In an ordinary transformer, all parameters work on every token. In a mixture of experts, the feedforward (MLP) block block is replaced by n parallel blocks (the experts) and a small gating network, the router, which scores the experts for each token and picks the top k. The output is the sum of those few experts, weighted by the router (Jiang et al., 2024, sec. 2.1):
y = Σᵢ G(x)ᵢ · Eᵢ(x), G(x) = softmax(TopK(x · W_g))
Experts not chosen are not computed: this is sparse activation.
Why. It decouples the size of a model from the cost of running it. Shazeer et al. (2017) inserted between recurrent layers a layer with thousands of experts and up to 137 billion parameters, multiplying capacity with only minor losses in efficiency. In Mixtral 8x7B (Mixture of experts (Mixtral)), each layer has 8 experts and the router picks 2 per token: 47 billion parameters in total, 13 billion active (Jiang et al., 2024). The idea is much older than transformers (Jacobs et al., 1991) and now appears in large open models such as DeepSeek-V3: 671 billion parameters, 37 billion active per token (DeepSeek-AI, 2024).
Costs. All experts must sit in memory even though each token uses few, and the router must be kept from sending everything to the same experts.
A misleading name. "Expert" suggests topic specialists: one for biology, another for philosophy. Mixtral's authors tested this and found no split by topic: the experts chosen for ArXiv papers, biology and philosophy were distributed very similarly. What they did find was syntactic regularity: in Python code, the word self or indentation tokens tend to go to the same expert (Jiang et al., 2024, sec. 5). It is a useful reminder for anyone thinking about Fodor-style modularity of mind: that an architecture has modules does not mean they divide the work the way a human org chart would.