The word comes from surveying. A bench mark was a reference point of known height on which a support, the bench, held the leveling staff; the figurative sense of "a standard to compare against" dates from the late nineteenth century (Harper, n.d.). The practice is alive: Argentina's leveling network has 33,892 pillars along roads and highways, all referred to an origin set in 1923 with the Mar del Plata tide gauge (Instituto Geográfico Nacional, n.d.). The metaphor is apt: a benchmark measures not absolute heights but differences from a shared fixed point.
How do you know whether one model is better than another? You can judge by vibes: use it, throw it a problem you think it can't solve, and be amazed when it does, which may be luck or exactly the kind of problem it was trained on. You can have experts evaluate it in their field, which is better but expensive and also fallible. You can open it up and look inside (interpretability), still slow and partial, and impossible when the weights are not public. Or you can give it standardized exams.
A benchmark is a standardized exam for machines: inputs whose correct answer is known, plus a way of scoring. Its virtues are obvious (it is reproducible, the same for everyone, cheap to run) and it has been one of the great engines of progress in AI, as the ImageNet competition showed (Chollet, 2019, sec. I.3.1). For language models, common choices include MMLU, with questions from 57 subjects (Hendrycks et al., 2021), GLUE, HellaSwag, or HumanEval for programming (Alammar & Grootendorst, 2024, pp. 374–375).
Millière & Buckner (2024b) list four problems (2024b, sec. 2.1): saturation (models overtake the human baseline ever faster), gamification (optimizing for the leaderboard; a case of Goodhart's law: when a measure becomes a target, it ceases to be a good measure (Alammar & Grootendorst, 2024, p. 377)), contamination, and a lack of construct validity: a test that measures a capacity in humans need not measure the same thing in a system with a different architecture.
The biggest risk is that of Clever Hans, the horse who seemed to do arithmetic but was responding to his questioners' involuntary cues (Pfungst, 1907/1911). A model can get things right through shortcuts: BERT reached 77% on an argument-comprehension task, close to the human baseline, but the result was explained by spurious cues in the data, and on an adversarial version it fell to chance (Niven & Kao, 2019). McCoy et al. (2019) showed that high-scoring natural language inference models failed on a dataset designed to defeat heuristics like "if every word of the hypothesis appears in the premise, it is an entailment." M. Mitchell (2019) describes the recurring recipe: define a narrow task, beat the human mark, and announce human-level performance on something much broader, such as "reading comprehension" (2019, ch. 13).
The critique has a flip side. If every test passed is declared insufficient, the goalposts move forever (Buckner, 2024, p. 281). Millière and Buckner point out that the distinction between competence and performance cuts both ways: a good score does not prove competence, but an error does not prove its absence either (Millière & Buckner, 2024b, sec. 2.1). Hence the proposals to measure the efficiency of acquiring skills rather than skill itself (see measuring intelligence) or to supplement behavior with mechanistic evidence. All of this is expensive, slow and thankless: there is far more money in building models than in evaluating them, and without good tests no one knows which lines of research are worth pursuing.