These are two assumptions in one. Independent: the occurrence of one example does not affect the others. Identically distributed: all are drawn from the same probability distribution. Thanks to them, the probability of the dataset factors into a product, p(x₁, …, xₙ; θ) = Πᵢ p(xᵢ; θ). The assumption rarely holds exactly, but often well enough to support useful inferences (Witten et al., 2017, p. 338). And only under this assumption does the error on a held-out test set estimate future error.
Independence fails. The words of a text are not independent: that is why a language model conditions each token on the previous ones (see Markov assumption), and the assumption applies to documents, not tokens. In reinforcement learning, the data depend on the agent's own actions.
Identity fails. The distribution of inputs may change while their relation to the output stays fixed (the covariate shift studied by Shimodaira, 2000), or the relation itself may change. M. Mitchell (2019) illustrates this with the "long tail": in 2016 there were reports that Tesla's Autopilot confused the salt lines laid on a highway before a snowstorm with lane markings (2019, ch. 6).
What an i.i.d. test set cannot see. If a shortcut is present in both the training and the test data, evaluation rewards it. A network meant to recognize animals learned to detect blurry backgrounds (M. Mitchell, 2019, ch. 6); natural language inference models such as BERT, excellent on MNLI, failed on HANS, a set where the lexical-overlap heuristic breaks down (McCoy et al., 2019).
Out of which distribution? Buckner (2024) notes that the claim that networks only generalize to i.i.d. data has become something of a mantra, rarely made precise: the distribution of ImageNet is not precisely characterized, and that of GPT-3 covers most publicly available written language (2024, p. 138). Saying that a case is out of distribution requires saying which distribution, and by what measure.