In classical machine learning, learning means changing parameters. In-context learning is the exception popularized by GPT-3: the model is shown, within the prompt, a few demonstrations of a task (or just a description) and completes the next case, without any weight update (Brown et al., 2020, sec. 2). Depending on the number of examples, one speaks of zero-shot, one-shot or few-shot prompting (Alammar & Grootendorst, 2024, pp. 180–181).
tall → short · hot → cold · full → ? (the model completes: empty)
Brown et al. (2020) describe it as the "inner loop" of meta-learning: during pretraining the model acquires skills and pattern-recognition abilities, and at inference time it uses them to recognize or adapt to the task. They cautiously add that the label is meant to stay agnostic on whether the model learns new tasks from scratch or recognizes patterns seen during training (2020, sec. 1). Buckner (2024) leans toward taking it seriously: GPT-3's performance with a few examples was far better than with one or none, which suggests that something is learned on the spot (2024, pp. 277–278).
How is this possible? Mechanistic interpretability has
proposed a partial mechanism, described by researchers at Anthropic (Elhage et al., 2021):
induction heads, attention heads that implement a rule of
the form [a][b] … [a] → [b], that is, "if I have seen B follow A, when A comes back I predict
B," sometimes in a more abstract version in which what is copied is only similar.
Buckner (2024) stresses the resemblance to the empiricist idea, from Locke and Hume, that
abstraction can arise from copying plus some similarity (2024, pp. 281–282). The
philosophical question is what counts as learning. If knowledge does not change but behavior
does, was there learning or just the interpretation of an instruction? And when a human learns
a rule from three examples, is that so different?