Pretraining is the stage in which a model learns "a bit of everything" before learning anything in particular. A language model is given a gigantic corpus (web pages, books, code) and a single task: predict the next token (next-token prediction) or, in models like BERT, reconstruct hidden tokens (masked language modeling). Since the correct answer is already in the text, no human labels are needed: this is self-supervised learning.
The recipe popularized by GPT-1 was "generative pre-training" on unlabeled text, followed by discriminative fine-tuning for each task (Radford et al., 2018). Today this stage takes most of the compute: Llama 2, for instance, was trained on about 2 trillion tokens (Alammar & Grootendorst, 2024, p. 26). The result is called a base or foundation model. It can continue texts but does not follow instructions: faced with a question, it may answer with another question, as if completing a questionnaire (Alammar & Grootendorst, 2024, pp. 355–357). Turning it into an assistant requires later stages: fine-tuning, instruction tuning and RLHF.
A common misunderstanding is to imagine that pretraining "loads" the data into the model, as into a database. What gets adjusted are parameters that compress regularities of the corpus; no index of documents remains to be looked up. This is why Santelli (2026) says these systems are grown rather than built: design fixes the architecture and the objective, but what the model ends up knowing arises from training. Buckner (2024) places this bet (more data and more compute, few built-in assumptions) at the empiricist end of the debate with nativism (2024, p. 25). The question remains open: how much of what a base model seems to "know" is knowledge of the world, and how much is knowledge of how people write about the world?