A mixture of experts is a way of making a network larger without making the computation larger: instead of a single feedforward network per layer there are several (“experts”), and a small router network decides which of them process each token. Mixtral 8x7B, by Mistral AI (weights released in December 2023, paper in January 2024), is the specimen that popularized the technique among open weights models: each layer has 8 experts and the router picks 2 per token, so the model has 47 billion parameters but uses about 13 billion per token (Jiang et al., 2024).
How it works
The expert layer replaces the feedforward block of each transformer block. For a token's vector x, the router computes scores x·W_g, keeps the two highest, applies a softmax to them, and the output is the weighted sum of the two chosen experts: y = Σᵢ softmax(Top2(x·W_g))ᵢ · SwiGLUᵢ(x) (Jiang et al., 2024, sec. 2.1). The rest is Mistral 7B's architecture: 32 layers, 4,096-dimensional vectors and grouped-query attention (GQA), in which several heads share keys and values (Jiang et al., 2024, sec. 2; Alammar & Grootendorst, 2024, pp. 98–100). Since attention and embeddings are not multiplied, “8x7B” does not add up to 56 billion.
The key distinction is between total parameters, which grow with the number of experts, and active parameters, which grow with how many are chosen per token (Jiang et al., 2024, sec. 2.1). Memory must hold all 47 billion (about 93 GB in bf16), but the compute per token is that of a 13-billion model.
Where it comes from
The idea is old: Jacobs et al. (1991) proposed “adaptive mixtures of local experts.” Shazeer et al. (2017) turned it into a sparse layer for deep networks: thousands of experts between LSTM layers, up to 137 billion parameters, and a gating network that picks a few per example, which they called conditional computation. They already noticed the central problem: the gating network tends to keep favoring the same experts, which therefore train faster and get chosen even more; they fixed it with an extra loss that rewards balance (Shazeer et al., 2017, sec. 4).
Compared with GPT-3, which is dense (it uses all 175 billion parameters for every token), Mixtral matches or beats Llama 2 70B and GPT-3.5 on most benchmarks, with clear advantages in math, code, and French, German, Spanish and Italian (Jiang et al., 2024, secs. 3–3.1). Its Instruct version was tuned with SFT and DPO. Since then, the sparse mixture has become standard at the frontier: DeepSeek-V3, the base of R1, has 671 billion total parameters and 37 billion active per token.
Limits
Everything must fit in memory, and spreading experts across several GPUs means shipping tokens from one to another and making sure none is overloaded (Jiang et al., 2024, sec. 2.1). The name, moreover, is misleading: Mixtral's routing analysis found no topic experts (see the professor's note). Mistral did not publish the size or composition of the data or the training compute: the data and compute base stats of this entry are estimates, not published figures.