Across dozens of layers, activation values can grow or shrink until training becomes unstable. Layer normalization corrects this token by token: it takes the d-component vector x, computes its mean μ and variance σ² over those d components, centers and rescales it, and then applies a learned gain γ and bias β:
LN(x) = γ ⊙ (x − μ) / √(σ² + ε) + β
(Ba et al., 2016). Example: the vector (2, 4, 6) has mean 4 and standard deviation ≈ 1.63; normalized, it becomes ≈ (−1.22, 0, 1.22), before γ and β.
Versus batch normalization. Batch normalization, common in vision, computes each unit's mean and variance across the examples of a mini-batch (Witten et al., 2017, p. 436). That depends on batch size and fits poorly with variable-length sequences. Layer normalization uses the statistics of a single example, performs the same computation in training and in use, and was designed partly with recurrent networks in mind (Ba et al., 2016).
Where it goes. In the original Transformer, after each residual sum:
LayerNorm(x + Sublayer(x)) (Vaswani et al., 2017, sec. 3.1). Since GPT-2 it has been
preferred before each sublayer (pre-normalization), as in GPT-3
(Brown et al., 2020, sec. 2.1); this shortens training (Alammar & Grootendorst, 2024, p. 101). Many current models,
such as LLaMA, use RMSNorm, which skips the centering and divides only by the root
mean square: simpler and cheaper (Alammar & Grootendorst, 2024, p. 101; Touvron, Lavril, et al., 2023, sec. 2.2).
Why it matters even when invisible. In diagrams it is often a small box ("Add & normalize" in BERT's), but it is one of the pieces that make depth trainable, together with residual connections. It contributes no "content": it fixes the scale at which the other layers can work.