The most basic rule of machine learning is not to evaluate on data the model has already seen: the test set must be kept apart from the training set, because any leak of information into the model ruins the measure of generalization. Evaluating on part of your training data is, says Chollet (2021), the worst thing you can do (2021, sec. 5.2).
With large language models, the rule is hard to follow. They are trained on a large slice of the web (see corpus), and benchmarks circulate on the web: they are copied, discussed, published with their solutions. The authors of GPT-3 built a "clean" version of each benchmark, without the examples that shared 13 consecutive words with the training data, and in most cases the score barely changed (Brown et al., 2020, sec. 4); in the extended version of the paper (arXiv:2005.14165) they also admit that a filtering bug left overlaps in and that retraining was too expensive. Detecting contamination is not trivial either: a paraphrase slips past literal string matching, and there is evidence that GPT-4 had seen solutions to competitive-programming problems and performed much worse on those absent from its training data (Millière & Buckner, 2024b, sec. 2.1). Sainz et al. (2023) distinguish degrees of contamination; the worst is a model that has seen the test split itself.
There is also a design dilemma. For a test to be repeatable and fair it should be published, but whatever is public ends up in the training data of the next models, and a lab can even train on it deliberately: the model becomes excellent at the benchmark without being good at what the benchmark meant to measure (see Goodhart's law and construct validity). One way out is to keep part of it secret, like ARC's private tasks (Chollet, 2019, sec. III.1.1), at the cost of transparency.
The result is a new kind of Clever Hans: the model does not read the examiner's gestures, it remembers the exam. Chollet (2019) gives the general version of the problem: with unlimited data on a task known in advance, one can "buy" skill without generalization (2019, sec. II.1.1). Contamination is the limiting case.
The boundary, though, is blurry. When is recognizing a problem similar to one already seen a matter of memory, and when is it learning? We ask the same of students (did they understand, or memorize the answers?); the difference is that with a student we know, roughly, which books they read.