The transformer is a neural network architecture that processes a whole sequence at once and lets every position "consult" all the others through attention. It was introduced in 2017 as a machine translation system and became the basis of nearly every language model in use today.
Built to translate
The starting problem was machine translation. The best systems of the time were recurrent encoder–decoders with attention (Bahdanau attention), the design Google had brought to its translator in 2016 (M. Mitchell, 2019, ch. 12). Vaswani and his seven coauthors proposed keeping the design but dropping recurrence: attention does all the work (Vaswani et al., 2017, sec. 1). They tested it on translation, English to German and English to French, using the sentence pairs of the WMT 2014 shared task (2017, sec. 5.1; Mollick, 2026); the arXiv version adds a constituency-parsing experiment. On English–German it beat everything published, ensembles included, by more than two BLEU points (2017, sec. 6.1).
Queries, keys and values
The central idea is self-attention. Each token enters as an embedding, and the model projects it, through three learned matrices, into three vectors (queries, keys and values) (Vaswani et al., 2017, sec. 3.2; Alammar & Grootendorst, 2024, pp. 91–94):
- the query says what information this position is looking for;
- the key says what this position offers so that others can find it;
- the value is what it actually contributes if another position picks it.
The dot product of one word's query with another word's key measures how relevant the second is to the first; a softmax turns those scores into weights that sum to 1, and the word's new representation is the average of the values weighted by them: softmax(QKᵀ/√dₖ)V. Dividing by √dₖ keeps the products of high-dimensional vectors from growing so large that the softmax saturates (Vaswani et al., 2017, sec. 3.2.1). Buckner (2024) notes that the terminology comes from key–value databases, with one difference: there is no explicit lookup of a record, only matrix operations that let each element modify the "meaning" of the others (2024, p. 272).
An example from Alammar & Grootendorst (2024): in The dog chased the squirrel because it…, continuing requires knowing what it refers to, and attention brings information from dog or squirrel into its representation (2024, pp. 88–89). (In Spanish, grammatical gender would do part of the job; English it does not help.) And it does this eight times rather than once: multi-head attention projects queries, keys and values eight times, each into a 64-dimensional subspace (512 ÷ 8), and combines the results (Vaswani et al., 2017, sec. 3.2.2). The authors observed heads that seemed to track the syntactic and semantic structure of the sentence (2017, sec. 4).
Why parallelism mattered
A recurrent network reads one word at a time: the state at position t depends on the state at t − 1, and that chain rules out parallelization within each example (Vaswani et al., 2017, sec. 1). In self-attention, the interactions among all positions are computed together as matrix multiplications, which is exactly what GPUs, chips born for video games, do best (Mollick, 2026). The base model trained in 12 hours on one machine with eight GPUs; the big one, in 3.5 days (Vaswani et al., 2017, sec. 5.2). The price: the cost of each layer grows with the square of the sequence length (2017, sec. 4).
The rest of the machine
Attention treats its input as a set (Raffel et al., 2020, p. 5), so a positional encoding made of sines and cosines is added to each embedding (Vaswani et al., 2017, sec. 3.5). Every sublayer is wrapped in a residual connection followed by layer normalization, and each block includes a feedforward (MLP) block network applied to each position separately (2017, secs. 3.1, 3.3). The encoder stacks six blocks; the decoder another six, with two differences: its self-attention carries a causal mask (it cannot look at what it has not generated yet), and it adds a cross-attention layer whose queries come from the decoder and whose keys and values come from the encoder, a direct heir of Bahdanau's attention (2017, sec. 3.2.3; Alammar & Grootendorst, 2024, pp. 15–17).
Offspring and limits
Like Eevee, it evolves according to the stone you bring near it: keeping the encoder gave BERT; keeping the decoder, GPT-1 and the GPT line; keeping both, T5 (Alammar & Grootendorst, 2024, pp. 18–21, 128). The paper's conclusion already announced images, audio and video (Vaswani et al., 2017, sec. 7); Vision Transformer (ViT) would follow. The original, however, was a specialist without pretraining, and its attention maps tell us what information is available, not how it is used (Buckner, 2024, p. 278; Millière & Buckner, 2024b, sec. 2.3).