Bender & Koller (2020) call form any observable realization of language: marks on paper, pixels or bytes, movements of the articulators. Meaning, by contrast, is a relation between form and something outside language: the pairs (e, i) of expressions and the communicative intents they can be used to evoke, mediated by the conventional meaning the linguistic system assigns to each expression. To understand is to retrieve i given e (2020, pp. 5186–5187). Hence the central thesis: a language model, trained only to predict strings, has in principle no way to learn that relation (2020, p. 5185).
The illustration is the The Deep-Sea Octopus: a hyperintelligent octopus taps the telegraph cable between two castaways, learns to predict one of them, and ends up impersonating him. He does fine in small talk, stumbles when asked for help building a coconut catapult, and fails when a bear shows up. If A found his messages meaningful, it was not because they made sense, but because A could make sense of them (2020, pp. 5188–5189). The authors grant that models learn "some reflection of meaning into the linguistic form," useful but incomplete (2020, p. 5193).
The replies target different premises (Millière & Buckner, 2024a, sec. 3.3). From conceptual role semantics, Piantadosi & Hill (2022) argue that much of lexical meaning lies in the internal relations among representations, which can be learned from distribution: this would be the inferential side of semantic competence, as opposed to the referential one. From externalism, Mandelkern & Linzen (2024) suggest that a model could refer by inheriting the causal chains of use of the community that produced its texts. And some doubt that text is "pure form" at all: it is full of descriptions, instructions, and code with unit tests, which the authors themselves recognize as partial grounding (Bender & Koller, 2020, p. 5193).
Note that the argument presupposes a Gricean notion of meaning. Bender and Koller read the slogan "meaning is use" as use to convey communicative intents to real people, not as distribution in a corpus (2020, p. 5191). An inferentialist would draw the line elsewhere: but where?