Hinton et al. (2015) open their paper with a biological image: many insects have a larval form optimized for feeding and an adult form optimized for moving and reproducing. Likewise, it pays to train a huge, cumbersome model to extract structure from data, and then transfer its knowledge to a small model fit for everyday use (2015, sec. 1).
The key lies in what gets imitated. A trained classifier does not just get the answer right: it also spreads probability over the wrong answers, and that spread is informative. A picture of a BMW has a tiny chance of being mistaken for a garbage truck, but that mistake is far more likely than mistaking it for a carrot. The student learns from these "soft targets" (the full distribution), not just from the correct answer. To make those tiny probabilities count, the temperature of the teacher's and the student's softmax is raised during training (2015, sec. 2).
The best-known case in language is DistilBERT: 40% smaller than BERT, 60% faster, and retaining 97% of its language-understanding performance (Sanh et al., 2019). The idea also travels among generative models: DeepSeek distilled the reasoning of its R1 model into smaller models and released them (Guo et al., 2025).
Distillation challenges an intuition that the authors themselves point out: we tend to identify a model's knowledge with the values of its parameters, which makes it hard to see how the form of the model could change while the knowledge stays the same. The alternative is to think of knowledge as a mapping from inputs to outputs, independent of its implementation (2015, sec. 1). It is an argument from multiple realizability in miniature: if two different networks compute almost the same function, do they know the same thing?