ElemedexA pocket encyclopedia of AI architectures
ENES
Chinchilla
No. 037

Chinchilla

Compute-Optimal

DecoderGenerative

It is four times smaller than its sibling Gopher, ate more than four times as much text, and beat it at almost everything on exactly the same budget. It never left the lab, but it named an era: the p(doom) song sings "Post-Chinchilla, super-dense."

Height
80 layers
Weight
≈ 140 GB (bf16)
Parameters
70 billion
Context
2048 tokens (as in Gopher)
Training data
1.4 trillion tokens (MassiveText)
Year
2022
Generation
5 · Large language models
Creators
Hoffmann, Borgeaud, Mensch, Buchatskaya, Cai et al. (DeepMind)
Kind
Model

Base stats

Parameters
72
Data
82
Compute
80
Context
40
Generality
65
Transparency
5
Total
344

Approximate, ordinal scales meant for comparing species with each other; the hard numbers are in the data sheet.

Weaknesses

  • Its weights were never released: the recipe traveled, the model did not.
  • The rule rests on only two comparable large runs (Chinchilla and Gopher) and on less than one pass over the data.
  • It optimizes the cost of training, not of using: it ignores that a model is queried millions of times.
  • It produces toxic language and stereotypes present in its corpus, as its own model card documents.

Habitat

It fed on MassiveText, DeepMind's corpus (web pages, books, news, code and Wikipedia), with proportions adjusted to reach 1.4 trillion tokens. Today it lives mostly as a rule of thumb in the spreadsheets of those who plan training runs, and in a line of the p(doom) song.

Moves

Abilities

Compute-optimal
With a fixed FLOP budget, parameters and tokens get equal shares: if the model doubles in size, the data should double too.
About 20 tokens per parameter
The paper's tables give about 20 training tokens per parameter for an optimal model; the industry's most quoted rule of thumb came from there.
Cheap inference
Being four times smaller than Gopher, it is cheaper to fine-tune and to run, for the same training cost.

Anatomy

Decoder block×80Decoder block×80same FLOPs (5.76 ×10²³)300B tokensEmbeddings, d = 16,384Causal attention, 128 headsFeedforward (4 × d)MMLU: 60.0%1.4T tokensEmbeddings, d = 8,192Causal attention, 64 headsFeedforward (4 × d)MMLU: 67.6%Gopher (280B parameters)Chinchilla (70B parameters)

One budget, two ways to spend it. Gopher and Chinchilla have the same depth (80 blocks), but Chinchilla is half as wide, has a quarter of the parameters and sees almost five times as many tokens. For equal training compute, it does better on almost every evaluation.

Evolution chain

  • compute-optimal: fewer parameters, far more tokens032 GPT-3
037 Chinchilla
  • smaller models, even more tokens, and released weights040 LLaMA

Chinchilla is a 70-billion-parameter language model that DeepMind presented in 2022. It is famous less for itself than for the rule it put to the test: if the compute budget is fixed, model size and amount of data should grow together (Hoffmann et al., 2022, sec. 1).

The question

A large model is trained only once, with a number of accelerators and an amount of time decided in advance. The practical question is how to split that budget between parameters and tokens. The scaling laws of Kaplan et al. (2020) answered: mostly on parameters. By their fit, optimal size grows as C^0.73 and data only as C^0.27; one should train huge models and stop them well before convergence (2020, secs. 1.1–1.2). Industry followed the recipe: GPT-3, Jurassic-1, Gopher and Megatron-Turing NLG, from 175 to 530 billion parameters, were trained on about 300 billion tokens (Hoffmann et al., 2022, table 1).

How they answered it

Hoffmann et al. (2022) trained more than 400 models, from 70 million to over 16 billion parameters, on 5 to 500 billion tokens, and estimated the optimal split by three different methods (2022, sec. 3). All three agreed: the exponents are roughly 0.5 and 0.5. Every time the model doubles, the data should double. Their tables give about 20 tokens per parameter (1 billion parameters, 20.2 billion tokens; 10 billion, 205 billion), the ratio that later became a rule of thumb (2022, table 3). The gap with Kaplan came partly from method: that work used the same learning-rate schedule for all training lengths, and smaller models (2022, sec. 2).

The conclusion implied that almost all large models were "undertrained." To test it, they trained Chinchilla on the same budget as Gopher (5.76 × 10²³ FLOPs): a quarter of the parameters and 1.4 trillion tokens, with the same 80-block decoder architecture but half as wide (2022, secs. 3, 4.1). Chinchilla outperformed Gopher (280 billion), GPT-3, Jurassic-1 and Megatron-Turing NLG on almost every evaluation and reached 67.6% on the MMLU benchmark, 7.6 points above Gopher (2022, secs. 4.2.2, 5).

What changed

Since then, data has become as important a bottleneck as parameters; the authors themselves call for larger, higher-quality datasets and warn about the toxicity and privacy issues that come with trillions of web-scraped tokens (2022, sec. 5). The recipe was soon adjusted again: the creators of LLaMA pointed out that Chinchilla's criterion ignores the cost of using the model, and trained smaller models for much longer; their 13-billion version beats GPT-3 on most benchmarks while being ten times smaller (Touvron, Lavril, et al., 2023, sec. 1). Alammar & Grootendorst (2024) register the effect on vocabulary: if a model ten times smaller does what GPT-3 does, is it still "large" (2024, p. 25)? The phrase "post-Chinchilla" made it into the p(doom) song, which uses it to mark an era (osmarks, n.d.).

Limits

The authors acknowledge that they have only two comparable large runs, that they assume a power law while there are signs of curvature, and that all their models saw less than one pass over the data (Hoffmann et al., 2022, sec. 5). The last point matters as available text runs out: with limited data, repeating it for up to four epochs is almost as good as new data, but beyond that the value of extra compute fades away (Muennighoff et al., 2025).

Sources

  • Alammar, J., & Grootendorst, M. (2024). Hands-on large language models: Language understanding and generation. O’Reilly Media. §
  • Buckner, C. J. (2024). From deep learning to rational machines: What the history of philosophy can teach us about the future of artificial intelligence. Oxford University Press. https://doi.org/10.1093/oso/9780197653302.001.0001 §
  • Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., . . . Sifre, L. (2022). Training compute-optimal large language models. In Advances in Neural Information Processing Systems (Vol. 35, pp. 30016–30030). Curran Associates. https://doi.org/10.52202/068431-2176 §
  • Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020). Scaling laws for neural language models (arXiv:2001.08361). arXiv. https://doi.org/10.48550/arXiv.2001.08361 §
  • Muennighoff, N., Rush, A. M., Barak, B., Le Scao, T., Piktus, A., Tazi, N., Pyysalo, S., Wolf, T., & Raffel, C. (2025). Scaling data-constrained language models. Journal of Machine Learning Research, 26(53), 1–66. https://jmlr.org/papers/v26/24-1000.html §
  • osmarks. (n.d.). P(Doom) song objectively correct interpretation. GTech™ Documentation. Retrieved September 30, 2026, from https://docs.osmarks.net/hypha/p(doom)_song_objectively_correct_interpretation §
  • Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., & Lample, G. (2023). LLaMA: Open and efficient foundation language models (arXiv:2302.13971). arXiv. https://doi.org/10.48550/arXiv.2302.13971 §