During pretraining, a model such as GPT-1 runs through a corpus and, at each position, tries to predict the next token from the k preceding ones. Training maximizes the sum of the log-probabilities assigned to the correct tokens, L = Σᵢ log P(uᵢ | uᵢ₋ₖ, …, uᵢ₋₁), where k is the size of the context window (Radford et al., 2018, sec. 3.1). Maximizing this quantity is the same as minimizing cross-entropy (see loss function).
Tiny example. The sentence “the cat sleeps on the bed” yields five free exercises: “the” → cat; “the cat” → sleeps; “the cat sleeps” → on; and so on up to bed. Nobody labels anything, because the answer is already in the text: hence self-supervised learning (Alammar & Grootendorst, 2024, p. 355). Thanks to the causal mask, a transformer solves all the exercises in a sentence in parallel.
In use, the same mechanism generates text: predict a token, append it to the input, and repeat (see autoregressive generation) (Alammar & Grootendorst, 2024, pp. 74–76). Unlike masked language modeling, the model only looks to the left.
The philosophical debate turns on the word “just”. Shanahan (2024) recommends remembering that an LLM, at the level of its operation, answers a single question: which words are likely to come next (2024, p. 70). But he also stresses that, while it is an engineering fact that this is what it does, the remarkable thing is that it suffices to solve previously unseen reasoning problems, if unreliably: the model does not cut and paste fragments but models an “unimaginably complex” distribution (2024, p. 77). Whether predicting well requires representing something of the world that produced the texts is the world model question. Describing the objective does not settle what capacities are needed to meet it.