Few words confuse a logician reading about AI as much as this one. Three senses should be kept apart:
- Logical. Going from premises to a conclusion; an inference is valid if it preserves truth, like modus ponens.
- Statistical. Estimating the unknown from data: population parameters, posterior distributions. "Probabilistic inference" computes, for instance, marginal probabilities in a Bayesian network (Witten et al., 2017, p. 368). Paradoxically, in this sense what a neural network does when it is trained (estimating its parameters from data) is inference.
- Engineering, the sense of this entry. In machine-learning jargon, "inference" is the opposite of training: using an already trained model to get predictions on new inputs (Chollet, 2021, sec. 3.6.7). It is, literally, running the forward pass through the network, without touching the weights.
In the engineering sense, inference has its own technical problems. Some layers behave differently than during training (dropout, for instance, is switched off) (Chollet, 2021, sec. 7.4.1). Its cost per token is about 2·N operations for a model with N parameters, versus about 6·N per training token (Kaplan et al., 2020, sec. 2.1), but it is paid every time someone uses the model. Hence the techniques for making it cheaper, such as quantization or the KV cache (Alammar & Grootendorst, 2024, p. 83), and the recent interest in deliberately spending more on it (test-time compute).
The confusion is not innocent. That a model "performs inference" on the sentence "All humans are mortal and Socrates is human, therefore…" does not mean it carries out a deduction: it computes which words tend to follow, and whether it lands on "Socrates is mortal" is a separate matter (Shanahan, 2024, p. 76). Whether a model draws inferences in the logical sense is an empirical question, studied with natural language inference tasks; whether it draws them in the normative sense of inferentialism is a philosophical one.