Instead of a single attention over full vectors, the transformer runs h attentions in parallel ("heads"), each with its own query, key and value matrices projecting into a smaller subspace; it then concatenates the results and projects them again (Vaswani et al., 2017, sec. 3.2.2):
MultiHead(Q,K,V) = Concat(head₁, …, headₕ)·W_O, headᵢ = Attention(Q·Wᵢ_Q, K·Wᵢ_K, V·Wᵢ_V)
The original transformer used 8 heads of 64 dimensions; BERT-base, 12; GPT-3, 96 heads in each of its 96 layers, that is, 9,216 (Vaswani et al., 2017, sec. 3.2.2; Devlin et al., 2019, p. 4173; Brown et al., 2020, sec. 2.1).
Why. A single softmax yields a single weighted average: if a token needs both its subject and its antecedent, the average blurs them. Several heads let the model attend to "different representation subspaces at different positions" (Vaswani et al., 2017, sec. 3.2.2). With one head, translation got 0.9 BLEU worse; with too many, it also degraded (Vaswani et al., 2017, sec. 6.2).
What each head might track. Take "The dog chased the squirrel because it was hungry."
One head might link it to dog (whose hunger?); another, each word to the previous one;
another, chased to its subject and object; another might "park" on a punctuation mark
when there is nothing useful to look at. Something like this was found in BERT:
K. Clark et al. (2019) found heads that track direct objects, determiners and
coreference, and others that mostly attend to delimiters; Buckner (2024) shows a head that
seems to resolve the anaphora in "she gave her a mint because her breath was bad"
(2024, pp. 278–279). Other heads copy patterns from context, [a][b] … [a] → [b]:
these are the so-called induction heads (Elhage et al., 2021; Buckner, 2024, pp. 281–282).
Warning. Nobody assigns these roles: they are the analyst's hypotheses. Visualizations are often hand-picked, many heads resist interpretation, and many can be pruned at little cost (Voita et al. (2019) removed 38 of 48 encoder heads for a loss of 0.15 BLEU). Moreover, the weights show where information came from, not what the model did with it (Buckner, 2024, p. 278).
Efficient variants (multi-query and grouped-query attention) share keys and values across heads to relieve the KV cache (Alammar & Grootendorst, 2024, pp. 98–100).