In self-attention, queries, keys and values all come from the same sequence: each token looks at every token of the same text (itself included) and is rewritten from what it finds (Vaswani et al., 2017, secs. 2 and 3.2.3). If X is the matrix with one vector per token:
Q = X·W_Q, K = X·W_K, V = X·W_V, output = softmax(Q·Kᵀ/√dₖ)·V
Row i of softmax(Q·Kᵀ/√dₖ) says how much token i "looks at" each of the others (Vaswani et al., 2017, sec. 3.2.1). See queries, keys and values.
Example. In "She deposited the check at the bank" and "They fished from the river bank", the word bank enters with the same vector; self-attention pushes it toward the financial region or the riverside region depending on whether it absorbs information from check or from fished and river. Buckner (2024) uses this very case, stressing that the cue may lie many words back (2024, pp. 270–271). This is how contextual embeddings arise.
Why it matters. In a recurrent network, information between two distant words must pass through as many steps as there are words in between; in self-attention, a single step connects any pair. This eases long-range dependencies and lets all tokens be processed in parallel (Vaswani et al., 2017, sec. 4), which is exactly what GPUs do well. The price: comparing every token with every other costs O(n²), so doubling the text quadruples the attention compute. Hence the limits of the context window and "sparse" variants such as the alternating layers of GPT-3 (Brown et al., 2020, sec. 2.1; Alammar & Grootendorst, 2024, p. 96).
Two regimes. In encoders like BERT it is bidirectional; in decoders like GPT-1 a causal mask prevents looking ahead (Alammar & Grootendorst, 2024, p. 97). In Vision Transformer (ViT) the "tokens" are image patches.
Caution. Vaswani and colleagues suggested that self-attention could yield more interpretable models (2017, sec. 4). But an attention map shows what information was available to a token, not how the model used it (Buckner, 2024, p. 278).