If a text says "…the cat sat on the mat. The cat…", an induction head notices that "cat" has occurred before, looks at what came next ("sat"), and raises the probability of repeating it (Millière & Buckner, 2024b, sec. 2.5.1). It is one of the most cited results of mechanistic interpretability, and one of the few circuits understood end to end.
The mechanism needs two layers (Elhage et al., 2021). In the first, a previous-token head writes into the residual stream, at [B]'s position, the information "the token before me is [A]." In the second, the induction head uses the current token as a query (queries, keys and values), finds the position whose "previous token" matches, attends to [B] and copies it into the prediction. It does not depend on learned statistics about which word follows which: it works even on sequences of random tokens.
Olsson et al. (2022) offered six lines of evidence that these heads may account for much of in-context learning. They appear at the very point in training where that ability jumps abruptly, visible as a bump in the loss curve. The evidence is causal in small attention-only models, and only correlational in large ones.
The philosophical question is what kind of thing was found. Millière & Buckner (2024b) see a non-content-specific computation: the previous-token head stores [A] in a subspace that works like a variable in a classical program, and the induction head reads it later, something like variable binding in symbolic systems (2024b, sec. 2.5.1). Buckner (2024) stresses something else: copying can be approximate, [A][B]…[A] → [B′], and copying enriched with abstract similarity yields something rule-like, as Locke and Hume had it (2024, pp. 281–282). Is this a victory for symbolism, which finds variables inside the network, or for empiricism, which sees rules born from copying?