For a distribution p₁, …, pₙ, entropy is H = −Σ pᵢ · log₂ pᵢ. It is 0 when one outcome is certain and maximal when all outcomes are equally likely; with base-2 logarithms it is measured in bits. Shannon introduced it in 1948 as a measure of “information, choice and uncertainty”, borrowing the name from statistical mechanics (Shannon, 1948, sec. 6); it can be proved to be the only function satisfying certain reasonable properties for measuring information (Witten et al., 2017, p. 110).
Tiny example. A fair coin: H = −(½ · log₂ ½ + ½ · log₂ ½) = 1 bit. A coin loaded 90/10: H = −(0.9 · log₂ 0.9 + 0.1 · log₂ 0.1) ≈ 0.47 bits. There is less uncertainty, because we almost always know what will come up.
Information gain is the entropy before learning an attribute minus the average entropy afterwards. It is the criterion decision trees use to choose what to ask first: in Witten et al.'s weather example, 9 “yes” days and 5 “no” days give 0.940 bits; after splitting on the outlook, 0.693 bits remain on average, so that question gains 0.247 bits, more than any other (Witten et al., 2017, pp. 107–110).
In language models entropy is everywhere. The loss function they are trained with is cross-entropy, and perplexity is its exponentiated version. Shannon himself estimated that the redundancy of English, counting dependencies of up to about eight letters, is roughly 50%; one of his methods was to delete letters from a text and ask someone to restore them (Shannon, 1948, sec. 7), a relative of masked language modeling.
But Shannon made clear on his first page that “these semantic aspects of communication are irrelevant to the engineering problem” (Shannon, 1948, p. 379). Shannon information measures improbability, not content: a source emitting random letters has more entropy per symbol than the English of a sonnet.