Fitting the training data is easy; what matters is getting new cases right. Chollet (2021) frames the central problem of the field as the tension between optimization (doing well on the training data) and generalization (doing well on data never seen before), and notes that only the former can be controlled directly (2021, sec. 5.1). That is why generalization is estimated empirically, on data the model has not touched (see training and test sets); its typical failure is overfitting.
The estimate rests on an assumption: that future data come from the same distribution as past data (data that are independent and identically distributed, i.i.d.). Witten et al. (2017) remind us that in real life things drift, and training data are always retrospective (2017, p. 505). Hence the distinction between in-distribution and out-of-distribution (o.o.d.) generalization: many critics argue that deep networks transfer well to the former and poorly to the latter (Buckner, 2024, p. 70). Chollet (2021) casts it as the difference between local generalization, by interpolation between nearby examples, and the extreme generalization humans achieve through abstraction and reasoning (2021, sec. 5.1.2); this is the basis of his proposal for measuring intelligence (Chollet, 2019).
The philosopher will recognize the problem of induction. Hume (1748/2007) showed that no past observation justifies, without circularity, the expectation that the future will resemble the past. Machine learning does not refute Hume: it manages him. It assumes uniformity (the i.i.d. assumption), half-tests it on a held-out set, and relies on an inductive bias to choose among the infinitely many generalizations consistent with the data. When nature changes course (a credit model trained on branches in New York and Florida and then applied to a new branch in Nebraska, in Witten and colleagues' example (2017, p. 163)), the assumption fails, and the model with it.