An n-gram model is the simplest kind of language model: it estimates the probability of the next word by looking only at the previous n − 1 words, and it does so by counting. It is the direct ancestor of the question we still put to large models: given this fragment, which words are likely to come next? (Shanahan, 2024, p. 70).
How it works
By the chain rule, the probability of a sentence is the product of the probabilities of each word given everything before it. The n-gram model simplifies this with the Markov assumption: only the last n − 1 words matter (a bigram conditions on one, a trigram on two). In the terms of Witten et al. (2017), it is a Markov model of order n − 1 (2017, pp. 403–404).
Training is counting: for a trigram, P(sat | the cat) = C(the cat sat) / C(the cat), the relative frequency in the corpus. To generate, you look up the context row, sample the next word, slide the window one step and repeat: autoregressive generation generation in its barest form. The model is evaluated by its perplexity on unseen text.
Markov, Shannon and a book opened at random
In 1913 Andrei Markov counted by hand the first 20,000 letters of Pushkin's Eugene Onegin to study how vowels and consonants follow one another (Markov, 1913/2006). Thirty-five years later, Claude Shannon carried the idea over to English with a "series of approximations" running from equiprobable letters through digrams, trigrams and, finally, words chosen according to the preceding word (Shannon, 1948, sec. 3). That last order already yields things like "THE HEAD AND IN FRONTAL ATTACK ON AN ENGLISH WRITER THAT THE CHARACTER OF THIS POINT…".
The sampling was done by hand: open a book at random, read until the current letter (or word) turns up, write down the one that follows, and carry on from another page. Shannon observed that going one order further would take an enormous amount of labor (1948, sec. 3), and called these sources "discrete Markoff processes" (1948, sec. 4). Meaning had been left out on purpose: semantic aspects, he writes, are irrelevant to the engineering problem (1948, p. 379).
Limits
The central problem is sparsity. With a vocabulary of V words there are Vⁿ possible sequences, and almost none appear in a real corpus; left uncorrected, a single unseen sequence zeroes out the probability of the whole sentence. Hence smoothing (the simplest scheme, the Laplace estimator, adds one to every count) is critical at higher orders (Witten et al., 2017, pp. 99, 404). The other answer was scale: Google released counts for about a billion five-word sequences drawn from a trillion-word corpus (2017, p. 404).
The second limit is myopia. A bigram model can learn that keys are is more frequent than keys is, but not the agreement in "The keys to the old, wooden kitchen cabinet are on the table," with six words between subject and verb (Mahowald et al., 2024). This was the target of Chomsky (1957): no finite-state model generates English, and statistical approximation does not define grammaticality (colorless green ideas sleep furiously) (1957, chs. 2–3). The second half of the argument aged worse: a class-based bigram model trained on newspaper text gives that sentence about 200,000 times the probability of its reversal (Pereira, 2000), a result Piantadosi (2024) turns against Chomskyan skepticism about probability (2024, pp. 368–369).
Offspring
The third limit is that each word is an isolated symbol. The neural probabilistic language model of Bengio et al. (2003) replaced counts with a network over learned word vectors, so that similarity could spread the evidence around; until then, n-grams had been one of the leading approaches (Alammar & Grootendorst, 2024, p. 88). LLMs inherited Shannon's task and changed almost everything else: unlike an n-gram model, GPT-2 mostly produces sequences of five or more words that do not occur in its training data (Mahowald et al., 2024).