The concept comes from psychometrics: Cronbach & Meehl (1955) called a construct the theoretical capacity a test aims to measure (intelligence, reading comprehension, anxiety) when no external criterion defines it, and construct validity the network of reasons that justifies reading the scores as measures of that capacity.
A score almost never matters in itself. A computer's result on a graphics benchmark matters because it predicts how the games you will actually play will run; a model's score on a math exam matters if it tells you something about what it can do that is not the exam. Teachers know the problem. An exam is not a report the teacher commissions in order to learn something: it serves to diagnose (and also to train), that is, to give us reasons to attribute to someone a competence, a disposition to solve similar problems in other circumstances. A driving test is not a mere formality: it grants a license because it attributes a competence, and with it a responsibility.
With AI systems the attribution is more fragile. A test designed for humans presupposes things about how answers are reached that may not hold for another architecture (Millière & Buckner, 2024b, sec. 2.1), and a model can succeed through shortcuts or contamination. Raji et al. (2021) criticize benchmarks presented as measures of "general" capabilities without a defined construct: a finite collection of tasks cannot stand for "everything in the whole wide world." Hernández-Orallo (2017) proposes instead a universal psychometrics that would hold for humans, animals and machines.
AGI makes the problem worse: one would first have to fix a construct of "general intelligence," and then tasks whose solution gives genuine reasons to attribute it; the kind of task where we would say "if it can solve this, I'll trust it with my finances." And what would the benchmark for a superintelligence look like? We might not notice.