The simplest way to learn is to memorize: store the examples and, faced with a new case, find the most similar one (Witten et al., 2017, p. 84); this is what k-nearest neighbors does. Deep networks do not store examples but weights, yet they have capacity to spare: trained long enough, they end up memorizing their data, and generalization appears at an intermediate point of training (Chollet, 2021, sec. 5.1.2) (see overfitting).
In language models, memorization became measurable. Carlini et al. (2021) extracted hundreds of verbatim sequences from GPT-2's training data, including names, phone numbers and email addresses, even though each appeared in a single document; larger models were more vulnerable. Carlini et al. (2023) later showed that memorization grows log-linearly with model size, with the number of times a text is duplicated, and with the length of the context supplied. Nasr et al. (2025) got ChatGPT, with a prompt that made it "diverge," to emit training data at a rate 150 times higher than normal.
Yet memorizing is not just a vice. Feldman (2020) argued that when a distribution has a long tail of rare cases, memorizing them is necessary to generalize well. And grokking shows both in sequence: first the network memorizes, then it forms a general circuit, and finally it cleans up the remnants of memorization (Millière & Buckner, 2024b, sec. 2.5.2).
Between those poles, Ted Chiang offered an influential image: an LLM is "a blurry JPEG of the web," a lossy compression that does not store the text verbatim but can reconstruct something like it (Chiang, 2023). The image explains at once what is preserved and what gets invented in interpolation (hallucination), although compressing well requires capturing regularities, which is precisely part of what we call generalizing.
Philosophy knows the extreme case: Blockhead, the table that would answer well without thinking. Millière & Buckner (2024b) take pure memorization as a null hypothesis and conclude that it cannot explain all LLM behavior, although any particular behavior might be approximate retrieval (2024b, sec. 4.2); hence also data contamination. But knowing that Paris is the capital of France is also having memorized it. Does the relevant difference lie between memorizing and generalizing, or between retrieving by lookup and reconstructing by rule?