Before attending, each token vector x is multiplied by three matrices learned in
training: q = x·W_Q (query), k = x·W_K (key) and v = x·W_V (value)
(Alammar & Grootendorst, 2024, p. 92). The attention of token i to token j is then:
score(i, j) = qᵢ·kⱼ / √dₖ → weights = softmax(scores) → outputᵢ = Σⱼ weightᵢⱼ·vⱼ
that is, a dot product between query and key, normalized, that weights the values (Vaswani et al., 2017, secs. 3.2 and 3.2.1; Alammar & Grootendorst, 2024, p. 94).
The database analogy. In a key–value table, a query that exactly matches a key returns its value. Here matching is graded: what comes back is a mixture of all the values, weighted by the similarity between the query and each key (Buckner, 2024, p. 272).
A teaching example. In "Sarah fed the cat because it…", we can imagine that the query of it encodes something like "who is my antecedent?", that the key of cat advertises "noun here, possible antecedent", and that its value carries what it should absorb about the cat, so that attention "bakes in cat information" (Alammar & Grootendorst, 2024, p. 91). These glosses are ours: learned vectors come unlabeled, and what they encode is found out by probing, not read off.
Details that matter. (1) Scores are divided by √dₖ because, for dₖ-dimensional vectors, dot products grow large and push the softmax into regions with almost no gradient (Vaswani et al., 2017, sec. 3.2.1). (2) Keys and values are separate because what makes a word relevant (its syntactic role, say) need not coincide with what it should contribute (its content). (3) On Buckner (2024)'s account, this three-way split, together with positional encoding, is what allowed recurrence to be dropped (2024, p. 272). (4) During generation, already computed keys and values are stored in the KV cache.
The vocabulary of "querying" and "looking for" is a handy metaphor; it should not be taken as attributing intentions to a matrix multiplication (see anthropomorphism).