A model's error on the data it was trained on is an optimistic, sometimes hopelessly optimistic, estimate of its future performance (Witten et al., 2017, p. 163). That is why data are held out. Standard practice uses three sets (Witten et al., 2017, p. 164; Chollet, 2021, sec. 5.2.1):
- Training: the examples used to fit the parameters.
- Validation: separate data for comparing configurations and choosing hyperparameters (how many layers, how many epochs).
- Test: data that play no part in any decision and are used at the end, to estimate generalization.
Why not just two? Because tuning the configuration by looking at validation performance is itself a form of learning: each decision leaks information about those data, and after many iterations the model ends up overfitted to the validation set without ever having seen it in training (Chollet, 2021, sec. 5.2.1). With little data, cross-validation is used: the set is split into k parts and each takes a turn as the test portion (Witten et al., 2017, p. 168).
The golden rule, "don't train on the test data," is broken in subtle ways. M. Mitchell (2019) recounts that in 2015 a Baidu team used at least thirty accounts to submit ImageNet results to the test server more than two hundred times, when the limit was two per week; that feedback may have bought it the fraction of a percentage point that decided the record, and it was disqualified (2019, ch. 5). With language models the problem changes shape: public benchmarks end up on the web and therefore in the training data (data contamination).
Philosophers of science will recognize the distinction between prediction and accommodation: a theory that gets right a datum not used to build it deserves more credit than one adjusted to accommodate it. The test set institutionalizes that intuition.