In machine learning, a world model is an internal representation that simulates aspects of the environment and predicts what would happen under an intervention: this is how reinforcement learning agents plan, by imagining the consequences of their actions (Millière & Buckner, 2024a, sec. 3.4). Applied to language models, the question changes shape: do they have representations of the entities and dynamics of the world that let them produce text consistent with it, or do they only handle surface statistics? This is more demanding than the question of reference: it is not enough for "cat" to pick out cats; the model must represent global properties of the situation in which discourse entities interact (2024a, sec. 3.4).
The most cited evidence comes from a board game. Othello-GPT is a GPT-style decoder trained only on sequences of moves, with no rules and no board. Even so, it proposes legal moves with high accuracy, and probes trained on its activations recover the state of the board; if those activations are edited to alter the state of a square, the predicted moves change accordingly (Li et al., 2023). Nanda et al. (2023) then showed that the board is encoded linearly, but relative to the player whose turn it is ("mine", "theirs", "empty") rather than by color (Millière & Buckner, 2024b, sec. 2.5.3).
One should not extrapolate too quickly. These are toy domains; networks prefer shortcuts as long as they keep lowering the loss; when Millière and Buckner wrote, interventionist evidence on the largest models was scarce, and their out-of-distribution failures suggested partial, brittle models. Moreover, whether a world model emerges depends on the text containing the relevant information and on the pressure of next-token prediction being strong enough to induce it; neither is guaranteed in advance (Millière & Buckner, 2024b, sec. 2.5.3).
The question matters because it offers an empirical way out of the Blockhead dilemma: a system that answers well because it represents the world is not a table of memorized replies (Millière & Buckner, 2024a, sec. 3.4). M. Mitchell (2019), by contrast, ties human mental models to an early intuitive physics and psychology and to the simulation of one's own experience (2019, ch. 14). Can a model learned from text alone count as a model of the world?