A language model does not write: it returns a probability distribution over the next token. The rule for going from that distribution to a concrete token is the decoding strategy (Alammar & Grootendorst, 2024, p. 79). The simplest is to always pick the most likely one (greedy decoding). Sampling, by contrast, is rolling a loaded die: if "Dear" has a 40% probability, it comes up 40% of the time (2024, p. 79). That yields variety, but also gives the tail of absurd options a chance. Two cutoffs control it (Alammar & Grootendorst, 2024, pp. 171–172):
- top-k: only the k most likely tokens are considered.
- top-p, or nucleus sampling: the most likely tokens are considered until their cumulative probability reaches p; the size of the nucleus varies with how confident the distribution is.
With the example from the temperature entry (logits 2, 1 and 0 for "car," "truck" and "elephant") and top-p = 0.9:
- T = 0.5 → probabilities 0.87 · 0.12 · 0.02 → nucleus: {car, truck}
- T = 1 → 0.67 · 0.24 · 0.09 → nucleus: {car, truck} (together 0.91)
- T = 2 → 0.51 · 0.31 · 0.19 → nucleus: all three
The elephant only enters the nucleus at high temperature; with top-k = 2 it would never get in. Within the nucleus, probabilities are renormalized: at T = 1, car gets 0.73 and truck 0.27. You can try it in the temperature lab.
Nucleus sampling was proposed by Holtzman et al. (2020), who showed something counterintuitive: searching for the most likely text produces bland, repetitive writing, very unlike human prose; human language is not the most probable sequence but a reasonable sample. Brown et al. (2020) used temperature 1 and top-p 0.9 to generate samples from GPT-3 (2020, app. F). Philosophically, the randomness of sampling complicates attributing an answer to "the model": the model is the whole distribution, and each answer is just one roll.