Chollet (2021) calls regularization the set of techniques that actively impede a model from fitting its training data perfectly, so that it performs better on validation data: the model becomes more "regular," its curve smoother (2021, sec. 5.4.4). It is the standard remedy for overfitting.
Penalizing the weights. The classic form adds a term to the loss function: loss = error + λ·R(w). With R = Σwᵢ² (the L2 norm) one gets ridge regression (Hoerl & Kennard, 1970), known in neural networks as weight decay (Witten et al., 2017, pp. 269, 399); with R = Σ|wᵢ| (the L1 norm), the lasso (Tibshirani, 1996). The gradient of w² is 2w, which shrinks along with the weight: L2 shrinks everything, but almost nothing reaches zero. The gradient of |w| is constant: L1 pushes small weights all the way to zero and thereby selects variables.
A Bayesian reading. Minimizing error + λ‖w‖² amounts to finding the most probable weights under a zero-centered Gaussian prior; a Laplace prior, with more mass at zero, yields the L1 penalty (Witten et al., 2017, pp. 393–394). λ measures how much the prior weighs. The preference for simplicity stops being a methodological maxim: it is an explicit probabilistic hypothesis (see inductive bias).
Other forms. Stopping training when the validation error starts to rise (early stopping) (Witten et al., 2017, p. 269); dropout; data augmentation, which adds shifted, rotated or cropped copies of each image (Witten et al., 2017, p. 437). Large language models tend to use little: GPT-3 was trained with a weight decay of 0.1, "a small amount of regularization" (Brown et al., 2020, app. B).
A puzzle. Zhang and colleagues (2017) showed that a convolutional network memorizes randomly assigned labels with or without explicit regularization. What makes deep networks generalize seems to lie elsewhere: in the architecture, the data, or some implicit regularization, perhaps by gradient descent itself. Buckner mentions this line of work (Soatto & Chiuso, 2014) but sets it aside as preliminary (2024, p. 119, n. 7).