At each step, a language model produces a score (logit) for every token in the vocabulary, and the softmax turns them into probabilities. Temperature T divides the logits before the softmax: p_i = exp(z_i / T) / Σ_j exp(z_j / T). With T < 1, differences are magnified and the favorite takes almost everything; with T > 1, they shrink and unlikely candidates gain ground. In the limit T → 0, the most likely token is always chosen (greedy decoding) (Alammar & Grootendorst, 2024, pp. 80, 171).
Take the example from Alammar & Grootendorst (2024): after "I am driving a…", "car" is more likely than "truck," and "elephant" is possible but rare (2024, p. 170). Suppose logits of 2, 1 and 0:
| T | car | truck | elephant |
|---|---|---|---|
| 0.5 | 0.87 | 0.12 | 0.02 |
| 1 | 0.67 | 0.24 | 0.09 |
| 2 | 0.51 | 0.31 | 0.19 |
At T = 0.5, "elephant" shows up in about 1 of every 60 samples; at T = 2, in almost 1 of every 5. You can turn the knob yourself in the temperature lab.
In practice, low temperatures suit tasks with one right answer (a formal email, a translation) and high ones suit brainstorming; temperature is combined with cutoffs such as top-p (sampling) (Alammar & Grootendorst, 2024, pp. 171–172). The name comes from statistical physics, where a formula of the same form (the Boltzmann distribution) describes how a hotter system spreads its energy over more states; Hinton et al. (2015) use the same knob for distillation.
A common misunderstanding is to call it the "creativity" knob. What changes is not what the model knows but how its distribution is sampled: the model proposes probabilities and the system around it decides. That raises a pragmatic question: if the same question, put to the same network, can receive different answers depending on a knob, who asserts what finally gets said, the model or the die?