A generative adversarial network (GAN) is a pair of networks trained against each other: a generator that turns random noise into data (typically images) and a discriminator that tries to tell real data from forgeries. Goodfellow et al. (2014) introduced it with an analogy that became canonical: the generator is a team of counterfeiters trying to pass fake currency, the discriminator is the police trying to catch it, and the competition improves both until the copies are indistinguishable from the genuine article (2014, sec. 1).
How it works
The generator G takes a vector z drawn from a simple distribution (its latent space) and turns it into an image G(z). The discriminator D receives images and returns the probability that they are real. Both are trained by gradient descent on a single value, in a minimax game (Goodfellow et al., 2014, sec. 3):
V(D, G) = E[log D(x)] + E[log(1 − D(G(z)))]
D tries to maximize it (to be right); G tries to minimize it (to deceive). In practice, a few steps of D alternate with one step of G. Because early on D rejects the forgeries easily and G's gradient vanishes, the authors recommend that G maximize log D(G(z)) instead. At the theoretical optimum, the generator's distribution matches the data distribution and D answers ½ everywhere (2014, sec. 4).
A philosophically curious detail: the generator never sees a real image. It only receives the gradients that reach it through the discriminator, so no example is copied directly into its parameters (2014, sec. 6).
What changed
Compared with its ancestor, the multilayer perceptron, the novelty is not the architecture (in 2014, G and D were plain MLPs trained on MNIST, a Toronto face database and CIFAR-10 (2014, sec. 5)) but adversarial training: the generator's loss function is another network that is also learning. Earlier generative models stumbled on intractable probabilistic computations and Markov chains; a GAN needs only backpropagation. Unlike the VAE, it has no encoder and represents no explicit distribution (2014, sec. 6); in exchange, its samples tend to be sharper.
Evolution was fast. DCGAN (Radford et al., 2016) made the generator convolutional: according to Buckner (2024), its layers perform near-inverses of the operations in a network like AlexNet, with transpose convolutions that go from abstraction to detail (2024, pp. 58, 207–209). DCGAN brought latent arithmetic: the vector for "man with glasses," minus that for "man," plus that for "woman," yields women with glasses (2024, p. 213). StyleGAN (Karras et al., 2019) was trained on 70,000 Flickr faces at 1024 × 1024, and a descendant of it powers sites like This Person Does Not Exist (2024, p. 212). Chollet devotes a section to implementing a simple GAN on celebrity faces, with a "bag of tricks" to keep it stable (Chollet, 2021, sec. 12.5).
Limits
- Mode collapse: if G is trained too much without updating D, it ends up sending many values of z to the same image. Goodfellow and colleagues called this "the Helvetica scenario" (Goodfellow et al., 2014, sec. 6).
- Instability: the balance between the two networks is fragile, and there is no likelihood that would measure progress clearly.
- Artifacts: transpose convolutions leave checkerboard patterns (Buckner, 2024, p. 210).
- Forgery: Buckner (2024) associates GANs with deepfakes (2024, p. 58), although the first face swaps, in 2017, used autoencoders with a shared encoder (Afchar et al., 2018).
Since the early 2020s, diffusion models have displaced it in much of image generation. The idea survives as an auxiliary loss: the autoencoder of latent diffusion is trained with an adversarial objective to avoid blurriness (Rombach et al., 2022, sec. 3.1).