A diffusion model generates data by learning to undo a gradual corruption. Take an image, add a little noise, then a little more, hundreds of times, until nothing of it is left; then train a neural network to walk that path backwards. The idea comes from nonequilibrium thermodynamics (Sohl-Dickstein and colleagues proposed it in 2015), but it was Ho et al. (2020) who showed that it could produce high-quality images (2020, sec. 1).
How it works
The forward process is a fixed Markov chain with nothing to learn: at each step a little Gaussian noise is added, with T = 1000 steps and variances growing from 10⁻⁴ to 0.02, until the image is indistinguishable from static (Ho et al., 2020, secs. 2, 4). The reverse process is learned. Ho and colleagues' practical finding was that the network is best trained with a simplified objective: given a noisy image and the step number, predict which noise was added (2020, secs. 3.2, 3.4). At bottom this is a denoising autoencoder trained at every noise level at once; the authors show that it is equivalent to a known technique, denoising score matching, which estimates the direction in which data density increases, and that generating amounts to following that slope with Langevin dynamics.
The network is a U-Net: a convolutional architecture that compresses the image and expands it again, with shortcuts between levels and self-attention layers at low resolution; the step number enters as a sinusoidal positional encoding, the same one used in the Transformer (2020, sec. 4, app. B). On CIFAR-10 the DDPM reached an FID of 3.17, better than almost every model published up to then (2020, sec. 4.1).
Latent diffusion and text
Working on pixels is expensive. Rombach et al. (2022) proposed running diffusion in the latent space of a pretrained autoencoder, which shrinks the image several times per side while keeping what matters perceptually, and adding cross-attention layers to condition generation on text or other inputs (2022, secs. 3, 3.3). That is the basis of Stable Diffusion, released with open weights in August 2022: according to the CompVis repository (2022), an 860-million-parameter U-Net conditioned by the frozen text encoder of CLIP, which runs on a 10 GB GPU. At each denoising step the model combines the current state of the image with the "semantic direction" supplied by the text vector (Millière & Buckner, 2024b, sec. 3.1.1; Alammar & Grootendorst, 2024, p. 265).
What changed
Compared with the variational autoencoder, which decodes in a single step, and with GANs and their unstable adversarial training, diffusion spreads the difficulty over many small steps, each of them easy. As likelihood-based models, they do not suffer the mode collapse of GANs (Rombach et al., 2022, sec. 1). The price is speed: in the DDPM, generating 128 images of 256 × 256 took about 300 seconds on a TPU v3-8 (Ho et al., 2020, app. B).
Limits
Sampling remains slower than with a GAN (Rombach et al., 2022, sec. 5). Models conditioned on contrastive encoders such as CLIP fail at binding attributes to objects, interpreting spatial prepositions, counting and handling negation; Millière & Buckner (2024b) trace much of this to the contrastive objective, which treats texts almost as bags of words (2024b, sec. 3.1.1). And what they learn mirrors their data: Stable Diffusion's own model card warns that it reproduces and amplifies the biases of a web-scraped corpus that includes adult material and mostly English captions.