The scaling hypothesis holds that, once an architecture that scales well has been found, such as the Transformer, it is enough to train ever larger networks, with more data and more compute, for ever more sophisticated behavior to emerge, including, in the limit, general intelligence (AGI). Its canonical statement is Gwern Branwen's (2020), written as GPT-3 appeared: the "blessings of scale" would make many problems of simple networks vanish once they are made very large. It is the optimistic, engineer's version of the bitter lesson (Sutton, 2019).
It rests on an empirical finding, scaling laws: a language model's loss falls as a power law in size, data and compute, with trends spanning more than seven orders of magnitude (Kaplan et al., 2020). Chinchilla corrected the recipe: for a fixed budget, parameters and tokens should grow together, and most large models had been undertrained (Hoffmann et al., 2022). The song records the era: "Post-Chinchilla, super-dense" and, without euphemism, "Hundred thousand GPU" (osmarks, n.d.).
In San Francisco, believing in scaling or not (being scale-pilled) defines camps more than any theory of mind does, and justifies gigantic investments in data centers (NVDA to the moon). The objections are of three kinds. Technical: loss is a proxy metric, and its decline does not guarantee new capabilities; emergent abilities might partly be a mirage of the metric (Schaeffer et al., 2023). Material: data runs out; Ilya Sutskever said at NeurIPS 2024 that "pre-training as we know it will end," because there is only one internet. Conceptual: a power law describes a trend, it does not explain what is learned; that the curve continues does not tell us whether it leads to understanding or to ever better imitation. The turn toward the test-time compute of reasoning models was read both ways: as a new axis of scaling, or as an admission that the old one was running out.