In AI jargon, "compute" is a quantity: how many multiplications and additions of decimal numbers are needed. Two uses of the acronym should be kept apart: FLOP, the total number of operations in a training run, and FLOP/s, the speed of a chip. A 2019 GPU such as the Titan RTX reached about 16 trillion single-precision operations per second (Chollet, 2021, sec. 1.3.1).
For transformers there is a rule of thumb: training costs about 6 operations per parameter per training token (2 in the forward pass, about 4 in backpropagation), that is, C ≈ 6·N·D (Kaplan et al., 2020, secs. 1.3, 2.1). For GPT-3: 6 × (175 × 10⁹ parameters) × (300 × 10⁹ tokens) ≈ 3.1 × 10²³ FLOP, which matches the figure in the original paper (Brown et al., 2020, app. D). Running the model, inference, costs instead about 2·N operations per generated token.
Compute matters for three reasons. It is one of the three axes of the scaling laws, together with parameters and data. It is expensive and concentrated, which is why it organizes the industry's economics (NVDA to the moon). And, being measurable, it has become a regulatory instrument: some rules use thresholds of 10²⁵ or 10²⁶ training FLOP as a proxy for risk (compute thresholds). The p(doom) song plays on the confusion between total and rate: "One E thirty flops a second / That was safe enough, we reckoned" (osmarks, n.d.).
Philosophically, measuring models in FLOP measures effort, not results. Chollet (2019) proposes looking at something else: intelligence as efficiency in turning priors and experience into skills (2019, sec. II.2.1). By that yardstick, a system that needs vastly more experience (and the compute to process it) to reach the same skill is not more intelligent but less (measuring intelligence).