The economist Charles Goodhart stated it in 1975, about British monetary policy: any observed statistical regularity will tend to collapse once pressure is placed on it for control purposes. The version in circulation ("when a measure becomes a target, it ceases to be a good measure") comes from the anthropologist Marilyn Strathern (1997), writing about university audits (Alammar & Grootendorst, 2024, p. 377). Donald Campbell had said something similar about social indicators.
In AI it is almost a law of nature, because training is optimizing a measure. Alammar and Grootendorst give a minimal example: if only grammatical sentences are rewarded, a model may learn to always say "This is a sentence." (2024, p. 377). Millière & Buckner (2024b) call the race for the top of benchmark leaderboards gamification: models optimized for the metric rather than for the capacity it was meant to measure (2024b, sec. 2.1). Reward hacking, sycophancy and data contamination are versions of the same phenomenon.
Manheim & Garrabrant (2018) distinguish four variants. Regressional: selecting on an imperfect proxy also selects its noise. Extremal: at extreme values of the proxy, its relation to the goal no longer holds. Causal: intervening on the proxy does not move the goal, just as heating the thermometer does not cure a fever. Adversarial: another agent, with other ends, exploits the proxy.
The underlying problem is one of construct validity: the relation between a measure and what it measures holds under certain conditions, and optimizing changes the conditions. That is why Chollet (2019) insists that skill at a task is not intelligence: with prior knowledge or unlimited data, skill can be "bought" (2019, sec. II.1.1) (see measuring intelligence). Is any measure immune to a sufficiently powerful optimizer, or does measuring intelligence presuppose a test-taker who is not trying to beat the test?