A diffusion model has two processes. The forward process is fixed and not learned: it takes an image x₀ and adds Gaussian noise in T small steps, until nothing of the signal is left (Ho et al., 2020, sec. 2). Thanks to a property of Gaussians, one can jump straight to any step t:
xₜ = √ᾱₜ · x₀ + √(1 − ᾱₜ) · ε, ε ~ N(0, I)
where ᾱₜ falls from nearly 1 to nearly 0. The reverse process is what is learned: a neural network receives a noisy image xₜ and the step number t and estimates the noise ε that was added. That is all training is: take images, corrupt them with known noise, and ask the network to guess it (Ho et al., 2020, sec. 3). To generate, one starts from pure noise and applies the network again and again, removing a little noise at each step. In Ho and colleagues' work, T = 1000 and the network is a U-Net (Ho et al., 2020, sec. 4).
Origin and kinship. The idea comes from non-equilibrium thermodynamics (Sohl-Dickstein et al., 2015); Ho et al. (2020) made it competitive in 2020 (Diffusion model). It shares with the VAE the notion of latent variables and with the GAN the goal of generating images, but it is simpler to train (2020, sec. 1): there are no two networks competing, just one learning to clean.
Latent diffusion. Diffusing directly over pixels is expensive. Rombach et al. (2022) moved the process into the latent space of a pretrained autoencoder and added cross-attention layers to condition the image on a text. This is how Stable Diffusion works, using CLIP's text representations to steer generation (Alammar & Grootendorst, 2024, p. 265).
A philosophical angle. Generating by "sculpting" out of noise invites comparisons with imagination, which Buckner (2024) discusses in connection with other generative models: moving from an abstract representation to a concrete exemplar by filling in plausible details (2024, pp. 205–211). The difference is that here detail is added gradually and the initial "sketch" is pure chance.