Kaplan et al. (2020) trained transformers of widely different sizes (from about a thousand to a billion parameters) and found that test loss (the cross-entropy of predicting the next token) depends mainly on three quantities: the number of parameters N, the dataset size D and the compute C. When neither of the other two is a bottleneck, the relationship is a power law, for instance L(N) = (N_c / N)^0.076, and the trends hold across more than seven orders of magnitude (2020, secs. 1.1–1.2). Architectural details such as the ratio of width to depth, by contrast, barely matter.
On a log–log plot that is a straight line, and a straight line invites extrapolation: how much compute the next improvement will take. Kaplan and colleagues concluded that, on a fixed budget, one should train very large models on relatively little data. Chinchilla corrected the recipe: parameters and tokens should grow in equal proportion. Chinchilla, with 70 billion parameters and 1.4 trillion tokens, about 20 tokens per parameter, outperformed models four times larger trained with the same compute (Hoffmann et al., 2022).
Three caveats. First, these are laws in a phenomenological sense, like Zipf's law, not derivations from first principles: they summarize inductions over training runs, and extrapolating them is a bet. Second, they predict loss, not capabilities; whether a specific skill appears suddenly or gradually is another debate (emergent abilities). Third, they fueled a radically empiricist reading of progress, the bitter lesson and the scaling hypothesis, on which more data and compute are all it takes. Buckner (2024) thinks that extreme reading is not the right lesson, even though scaling research is what fuels it (2024, p. 25).