Technically, a language model is a probability distribution over sequences. By the product rule, that probability decomposes into conditional probabilities: P(w₁ … wₙ) = P(w₁) · P(w₂ | w₁) · … · P(wₙ | w₁ … wₙ₋₁). Modeling language thus means estimating the probability of the next token given a context (Buckner, 2024, p. 268).
Tiny example. A bigram model estimates P(cat | the) by counting how often “the” is followed by “cat” in a corpus; n-gram models generalize the idea to contexts of n − 1 symbols (Witten et al., 2017, pp. 403–404). This is how Shannon, in 1948, generated increasingly plausible “approximations to English”, first with letters and then with words (Shannon, 1948, sec. 3). It is the n-gram language model species; neural models replaced counts with learned vectors (Bengio et al., 2003), and transformers stretched the context to thousands of tokens.
The phrase is ambiguous. It can name the mathematical object (the distribution), the trained network that approximates it, or, by extension, the conversational product built on top: ChatGPT is a product, not a model (Alammar & Grootendorst, 2024, p. 23). Shanahan (2024) asks us to keep apart the “bare-bones” LLM, which answers the question of which words are likely to come next, and the systems in which it is embedded (2024, p. 70). Moreover, encoders such as BERT are also called “language models”, although they predict hidden words rather than the next token (masked language modeling) and do not directly assign a probability to a whole sentence. And “large” is arbitrary: what is large today may be small tomorrow (Alammar & Grootendorst, 2024, p. 25).
In linguistics the topic has a history. Colorless green ideas sleep furiously was offered for decades as evidence that syntax is autonomous and that transitional statistics between words are not enough (Chomsky, 1957); Piantadosi (2024) argues that this claim has been refuted and that current models undercut the principles of the generative program (2024, pp. 367–368). See formal vs functional competence.