Perplexity evaluates how well a language model predicts a real text. The model is asked, token by token, what probability it gave to what actually came next; the surprisal of each hit (−log₂ p) is averaged, and the result exponentiated. If the model gives high probability to each correct token, it is not “perplexed” (Alammar & Grootendorst, 2024, p. 374). As a formula, PPL = 2^H, where H is the average cross-entropy, in bits per token.
Tiny example. Three tokens to which the model assigned probabilities 1/2, 1/4 and 1/8. Their surprisals are 1, 2 and 3 bits; the average is 2 bits; the perplexity is 2² = 4. It is read as an effective branching factor: on average, the model is as undecided as if it were choosing among 4 equally likely options.
The notion comes from 1970s speech recognition (Jelinek et al., 1977) and is the natural metric for pretraining, because it measures the very quantity being optimized (see next-token prediction). The creators of GPT-2, for example, tuned each model's learning rate by looking for the best perplexity on a held-out portion of their corpus (Radford et al., 2019, sec. 3).
Limits. It depends on the tokenizer: a model with longer tokens makes fewer predictions per text, so perplexities of models with different vocabularies are not directly comparable; that is why results are often rescaled to a canonical unit such as the character, the byte or the word (Radford et al., 2019, sec. 3.1). And it measures fit to a corpus, not truth or usefulness: it says nothing about the consistency, fluency, creativity or correctness of what the model generates (Alammar & Grootendorst, 2024, p. 374). A very typical error can have low perplexity; a truth phrased in an unusual way, high perplexity.