A VAE is an encoder–decoder that learns to compress images (or other data) into a low-dimensional latent space and, above all, to come back from that space to plausible exemplars. What sets it apart from an ordinary autoencoder is that it encodes each input not as a point but as a probability distribution. Kingma & Welling (2014) proposed it in December 2013; Rezende et al. (2014) reached an equivalent idea independently (2014, sec. 4).
How it works
The model has three parts (Kingma & Welling, 2014, sec. 3):
- An encoder takes an image x and returns two vectors, means μ and standard deviations σ. Together they describe a Gaussian bell in latent space: the region where the code for x probably lives.
- A code z is sampled from that region. So that the random draw does not cut the flow of gradients, the model uses the reparameterization trick: z = μ + σ ⊙ ε, with ε drawn from N(0, I). Randomness enters as an external input, and backpropagation can pass through μ and σ (2014, sec. 2.4).
- A decoder turns z back into an image.
The loss function (the negative of the "variational lower bound") has two terms: a reconstruction error and a Kullback–Leibler divergence that penalizes each code's distribution for straying from N(0, I). The authors read the second term as a regularizer (2014, sec. 2.3). Once training is over, the encoder can be thrown away: draw a point from N(0, I), decode it, and a new exemplar appears.
The original experiments were modest: perceptrons with a single hidden layer (500 units for MNIST, 200 for about two thousand frames of Brendan Frey's face) and latent spaces of 2 to 200 dimensions (2014, sec. 5).
What changed
An ordinary autoencoder assigns each image a point and leaves the rest of the space unused: between two codes there are "holes" that decode into garbage. The problem is not trivial. Chollet (2021) points out that the pixel-by-pixel average of two MNIST digits is usually not a digit, and that meaningful interpolation requires moving along the latent manifold of the data (2021, sec. 5.1.2). The KL term forces the codes to overlap near the origin, so the space is filled: any path between two points yields gradual, plausible transformations. Buckner (2024) describes this as a "smoother" latent space, in which means and standard deviations separate the factors of variation and let them be sampled coherently (2024, pp. 215–217). Chollet devotes a section to implementing it in Keras, with "concept vectors" for image editing (2021, sec. 12.4).
The other change is one of lineage: the VAE joined deep learning to the Bayesian tradition of latent-variable models. GANs, born six months later, gave up that inheritance: they represent no distribution explicitly (Goodfellow et al., 2014, sec. 6).
Forms and offspring
Its most widespread form today is invisible. Latent diffusion models do not work on pixels but on the latent space of an autoencoder; one of the variants studied imposes a slight KL penalty "similar to a VAE" and is also trained with an adversarial loss to avoid blurriness (Rombach et al., 2022, sec. 3.1). In Stable Diffusion, the public release of that work, every image leaves through a decoder of this kind at the very end.
Limits
- Blurriness: with pixel-by-pixel losses, samples come out soft and out of focus, which is exactly the defect that the adversarial loss in latent diffusion tries to fix.
- Tension between the terms: more weight on the KL term gives a tidier space but worse reconstructions; variants such as β-VAE tune this balance to obtain more interpretable factors (Buckner, 2024, p. 217).
- Opaque axes: nothing guarantees that each latent dimension matches a feature a human could name.