A diffusion language model generates text without writing it left to right. It starts from a masked canvas and, step by step, predicts every position at once, keeps the most confident ones and masks the rest again. LLaDA showed that the recipe works at the scale of a large language model: 8 billion parameters trained from scratch on 2.3 trillion tokens. Its authors find it competitive overall with LLaMA 3 8B, which saw 15 trillion (Nie et al., 2025, secs. 2.2, 3.2).
How it works
The forward process is the discrete version of Diffusion model's noise: draw a t between 0
and 1 and replace each token by [MASK] with probability t; at t = 1 nothing is left. The
network that learns the way back is a Transformer without a causal mask that
predicts every masked token at once. The loss is computed on those tokens
only and is an upper bound on the negative log-likelihood: training on it is approximate
maximum-likelihood estimation, as for an autoregressive model (Nie et al., 2025, sec. 2.1).
So far, this is nearly BERT. The difference is that BERT always masks 15%, whereas here the rate varies between 0 and 1 and the loss is weighted by 1/t; on the authors' account, that subtle difference is what turns the gap-filling exercise into a generator (2025, sec. 2.1). The lineage is short: Austin et al. (2021) extended diffusion to discrete states and saw that an "absorbing" state (the mask) links diffusion to masked models; Lou et al. (2024) reached GPT-2's perplexity; Sahoo et al. (2024) showed that the objective reduces to a mixture of classical masked language modeling losses.
To answer, the model receives the prompt and a fixed-length, fully masked canvas. At each step it predicts every mask, keeps the most confident predictions and masks the others again (Nie et al., 2025, sec. 2.4). In masked diffusion, a token once placed stays placed.
What changed
Autoregressive generation takes one pass per token and stores what it has already computed in a KV cache; LLaDA, having no causal mask, cannot, and reprocesses everything at each step (Nie et al., 2025, sec. 2.2).
| Autoregressive | Diffusion | |
|---|---|---|
| Order | left to right | any |
| Tokens per pass | one | one or several, depending on the steps |
| KV cache | yes | no, in the pure form |
| Length | the model decides | a cap set in advance |
Here the number of steps is a hyperparameter: fewer steps buy speed at the cost of quality (2025, sec. 2.4). Since every prediction sees both sides, filling a gap in the middle comes naturally. And there is a further effect: Berglund et al. (2024) showed that an autoregressive model that has learned "A is B" does not automatically generalize to "B is A"; on a test with lines of Chinese poems, LLaDA recalls the preceding line almost as well as the following one (45.6% against 51.8%), whereas GPT-4o drops from 82.7% to 34.3% (Nie et al., 2025, sec. 3.3).
Industry went for speed. Google showed Gemini Diffusion as a demo in May 2025 (Google DeepMind, 2025); according to Inception's report, measured by Artificial Analysis, Mercury Coder Mini exceeds 1,000 tokens per second on H100 GPUs (Inception Labs et al., 2025). Dream 7B started from the weights of an autoregressive model (Ye et al., 2025), and LLaDA-MoE added experts (Zhu et al., 2025). Block diffusion generates blocks left to right and diffuses within each one, which brings back the KV cache and variable length (Arriola et al., 2025). DiffusionGemma (June 2026) combines both: a causal encoder processes the prompt and caches it, and a bidirectional decoder denoises 256-token canvases one after another; strictly, it is the same network in two modes. Instead of masks it uses random tokens, which can be redrawn, so it can correct what it has already written (Google DeepMind, 2026; Google, 2026; O'Donoghue & Flennerhag, 2026).
Limits
- Quality. DiffusionGemma's model card puts it below the same-size Gemma 4 on almost every test (77.6% against 82.6% on MMLU Pro), and Google recommends the autoregressive model when quality matters (Google DeepMind, 2026; O'Donoghue & Flennerhag, 2026).
- Conditional speed. The advantage holds for one user on one card; on a server that batches many requests, it fades (O'Donoghue & Flennerhag, 2026).
- Length and compute. Length is a hyperparameter, and earlier work cited by Nie et al. (2025) estimated that masked diffusion needs 16 times more compute to match an autoregressive model's likelihood; the authors reply that likelihood is only an indirect measure of performance (2025, secs. 3.1, 5).
- Alignment. LLaDA got supervised fine-tuning only, with no reinforcement learning (2025, sec. 3.2).