P(A | B) reads “the probability of A given B”: we restrict attention to the cases in which B occurs and ask in what proportion of them A occurs as well. Formally, P(A | B) = P(A, B) / P(B), provided P(B) > 0. Rearranging gives the product rule, P(A, B) = P(A | B) · P(B), from which Bayes' theorem follows in one line (Witten et al., 2017, p. 337).
Tiny example. With a die, P(6 | even) = P(6 and even) / P(even) = (1/6) / (1/2) = 1/3. Learning that the number is even doubled the probability of a 6, from 1/6 to 1/3.
Language models are, quite literally, conditional-probability machines: they estimate P(next token | previous tokens). A bigram model does it by counting: if “of” occurs 100 times in a corpus and “of the” 25 times, it estimates P(the | of) = 25/100 = 0.25. Markov and n-gram models (n-gram language model) condition only on the last n − 1 symbols (Witten et al., 2017, pp. 403–404); a transformer conditions on its whole context window.
A caveat for logicians: P(A | B) is not the probability of the material conditional B → A. With the die, P(even → 6) = P(odd or 6) = 4/6 = 2/3, whereas P(6 | even) = 1/3. And the tempting idea that the probability of a natural-language conditional is always the corresponding conditional probability runs into David Lewis's (1976) triviality results. How LLMs reason with conditionals and modals is now a research topic in its own right (Holliday et al., 2024).