A probability distribution says how likely each value of a random variable is. If the variable is discrete (a card, a word), each outcome gets a number between 0 and 1 and they all add up to 1; if it is continuous (a height), one uses a density and sums become integrals (Witten et al., 2017, pp. 336–337).
Tiny example. A fair die: P(1) = P(2) = … = P(6) = 1/6, and 6 · 1/6 = 1. A language model facing “I am driving a…” assigns something like {car: 0.62, truck: 0.30, bus: 0.07, elephant: 0.01}. The numbers are made up, but the shape is exact: elephant does not get probability zero, just a very low one (Alammar & Grootendorst, 2024, p. 170).
That is what an LLM really outputs: not a word, but a distribution over its entire vocabulary (32,064 tokens in the case of Phi-3), computed at the end of each forward pass by the language-modeling “head” (Alammar & Grootendorst, 2024, pp. 77–79). The softmax function turns arbitrary scores, the logits, into a valid distribution (Chollet, 2021, sec. 4.2); then a decoding strategy picks a token: the most probable one, or one drawn according to the probabilities (sampling, modulated by temperature) (Alammar & Grootendorst, 2024, pp. 79–80). Training the model, in turn, means bringing that distribution closer to that of real texts; the gap between the two is measured with cross-entropy (see entropy).
A philosophical caution. That a model “assigns probabilities” does not imply that it has degrees of belief. These are outputs of a function trained to fit the frequencies of a corpus; whether they can also be read as the credences of anything like a subject, or which interpretation of probability (frequentist, subjective, dispositional) suits them, is one of the open questions about these systems.