ElemedexA pocket encyclopedia of AI architectures
ENES

Glossary (TM) › Architectural parts

TM083

Discrete diffusion

Also: discrete-state diffusion, D3PM

In Spanish: Difusión discreta

Diffusion over the symbols of a finite vocabulary: noise is not added but replaces tokens, either with a mask or with other tokens drawn at random.

Diffusion over images (Diffusion model) works with real numbers: a little Gaussian noise is added to each pixel and the network learns to subtract it. With text there is nothing to add: a token is an element of a finite vocabulary, and "cat + 0.1" is nothing at all. Discrete diffusion replaces addition with a transition probability: at each step, each token stays put or jumps to another symbol according to a matrix Qₜ (Austin et al., 2021, sec. 3). What is learned is still the way back.

Austin et al. (2021) (D3PM) studied several matrices; two matter for text (2021, sec. 3.1):

  • Absorbing noise. Each token either stays or turns into [MASK], which it never leaves; in the end everything is masked, and the model knows which positions are missing. On text this variant performed best (2021, sec. 5), in SEDD as well (Lou et al., 2024, sec. 5.2). Sahoo et al. (2024) (MDLM) showed that its objective is a weighted average of masked language modeling losses and that, on the way back, in their parameterization, a revealed token never changes again (2024, secs. 3.2.3, 3.5). This is LLaDA's recipe (Nie et al., 2025, sec. 2.1).
  • Uniform noise. Each token can jump to any other; in the end there is a soup of random tokens. The model does not know which ones are wrong, but nothing is ever fixed: DiffusionGemma uses this variant so that it can correct what it has already written (Google, 2026).

The framework covers familiar models. According to Austin et al. (2021), BERT is a one-step diffusion, and an autoregressive generation model is a deterministic diffusion that masks one token per step, starting from the end (2021, sec. 4). In this framework, writing straight through and filling gaps are two ways of ordering the noise.

A detail for linguists: the authors also tried noise that jumps more often between tokens with similar embeddings (the idea: in a vocabulary of letters, vowels resemble one another more than they resemble consonants). It barely improved on uniform noise, a negative result that surprised them and that, they say, suggests that not every notion of structure is meaningful (2021, secs. 3.1, 5.1). Which similarity between signs would be the relevant one for corrupting a text?

Sources

  • Austin, J., Johnson, D. D., Ho, J., Tarlow, D., & van den Berg, R. (2021). Structured denoising diffusion models in discrete state-spaces. In Advances in Neural Information Processing Systems (Vol. 34, pp. 17981–17993). Curran Associates. https://proceedings.neurips.cc/paper/2021/hash/958c530554f78bcd8e97125b70e6973d-Abstract.html §
  • Google. (2026). Diffusion in text generation explained. Google AI for Developers. Retrieved October 12, 2026, from https://ai.google.dev/gemma/docs/diffusiongemma/explained §
  • Lou, A., Meng, C., & Ermon, S. (2024). Discrete diffusion modeling by estimating the ratios of the data distribution. In R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, & F. Berkenkamp (Eds.), Proceedings of the 41st International Conference on Machine Learning (Vol. 235, pp. 32819–32848). PMLR. https://proceedings.mlr.press/v235/lou24a.html §
  • Nie, S., Zhu, F., You, Z., Zhang, X., Ou, J., Hu, J., Zhou, J., Lin, Y., Wen, J.-R., & Li, C. (2025). Large language diffusion models. In Advances in Neural Information Processing Systems (Vol. 38, pp. 56354–56392). Curran Associates. https://doi.org/10.52202/085713-1689 §
  • Sahoo, S. S., Arriola, M., Schiff, Y., Gokaslan, A., Marroquin, E., Chiu, J. T., Rush, A., & Kuleshov, V. (2024). Simple and effective masked diffusion language models. In Advances in Neural Information Processing Systems (Vol. 37, pp. 130136–130184). Curran Associates. https://doi.org/10.52202/079017-4135 §