The trick is to manufacture a supervised problem without a supervisor. Take a piece of data, hide or corrupt part of it, and train the model to reconstruct it: the correct answer was already in the data. Chollet (2021) defines it as the case in which targets are obtained from the inputs themselves (2021, sec. 7.4). Examples:
- Next-token prediction: given "The cat sat on the," predict "mat." It is the pretraining task of GPT-1 and its successors; Alammar & Grootendorst (2024) present it as a self-supervised method, one without labels (2024, p. 355).
- Masked modeling: covering words in the middle of a sentence and reconstructing them, as BERT does (Devlin et al., 2019, p. 4174).
- Contrastive learning: learning which pairs belong together, for instance an image and its caption in CLIP.
Its practical importance is enormous. Since it requires no human annotation, it makes web-scale text usable: a corpus of trillions of tokens (Llama 2 was trained on 2 trillion (Alammar & Grootendorst, 2024, p. 26)) becomes trillions of exercises that grade themselves. This is the basis of the pretraining of large language models, and the reason human data annotation has been pushed to later, smaller stages such as fine-tuning or RLHF.
For the philosopher, the word "supervised" deserves suspicion. Nobody judges here whether a prediction was correct in any normative sense: the only yardstick is which word actually appeared in the text. The corpus acts as teacher, but a teacher who knows only what was said, not what ought to have been said. Whether that is enough to acquire something like meaning is one of the deep questions about language models (see form vs meaning).