A transformer is, above all, a stack of blocks identical in shape (with different weights). Each block has two sublayers: (1) multi-head attention, which moves information between positions, and (2) a feedforward (MLP) block network, which processes each position separately (Vaswani et al., 2017, sec. 3.1; Alammar & Grootendorst, 2024, p. 86). Each sublayer is wrapped in a residual connection and a layer normalization. In the original version:
x ← LayerNorm(x + Attention(x)), x ← LayerNorm(x + FFN(x))
All sublayers return vectors of the same dimension they receive (512 in the original), which is what makes stacking possible (Vaswani et al., 2017, sec. 3.1). The original decoder blocks add a third sublayer of cross-attention.
How many. Six on each side in the Transformer; 12 in GPT-1 and BERT-base; 96 in GPT-3 (Radford et al., 2018, sec. 4.1; Devlin et al., 2019, p. 4173; Brown et al., 2020, sec. 2.1). In Pokédex diagrams, that number is the specimen's "height".
How it changed. The 2017 block survives almost intact, with tweaks: normalization moves
before each sublayer (pre-norm: x ← x + Attention(LN(x))), which stabilizes training;
LayerNorm is replaced by RMSNorm; the feedforward network's ReLU, by SwiGLU; and attention
gains rotary positions and grouped queries (Alammar & Grootendorst, 2024, pp. 101–102; Touvron, Lavril, et al., 2023, sec. 2.2). In mixture of experts models, the feedforward network is
multiplied into several "experts" (Mixture of experts (Mixtral)).
How to think about it. There are two complementary readings. One is an assembly line: each block transforms the representation and hands it on, like the "abstraction sandwiches" that Buckner (2024) describes for convolutional networks (2024, p. 127). The other, favored by mechanistic interpretability, sees a single residual stream running through the stack, which every sublayer reads from and writes to (Elhage et al., 2021). The first suggests hierarchy; the second, a shared workspace. Neither is compulsory: they are two ways of looking at the same sum of matrices.