RLHF requires millions of evaluations, and no company can pay people to read them all. The solution is to train a second model that predicts human judgment (Alammar & Grootendorst, 2024, p. 379). One takes a copy of the instruction-tuned model and swaps its "head": instead of probabilities over the vocabulary, it returns a single number. InstructGPT used 6-billion-parameter reward models (Ouyang et al., 2022, sec. 3.5).
The data are comparisons: a prompt, an accepted response and a rejected one (not necessarily a good one and a bad one; sometimes both are good and one is better) (Alammar & Grootendorst, 2024, pp. 380–382). Training pushes the accepted response to score higher. The difference in scores is read as a probability of preference, following the Bradley–Terry model (1952) or the Elo system in chess (Christiano et al., 2017, sec. 2.2.3):
P(A preferred to B) = 1 / (1 + e^−(r_A − r_B)). If r_A − r_B = 1, P ≈ 0.73.
The risk is that the optimized model learns to please the judge rather than do what the judge is trying to measure (so-called Goodhart's law). It is the linguistic version of the reward hacking that Buckner (2024) documents in reinforcement learning: an agent that circles endlessly to collect bonuses without ever finishing the race (2024, p. 68). This is why InstructGPT penalizes drifting away from the previous model (Ouyang et al., 2022, sec. 3.5), and why Llama 2 uses two reward models, one for helpfulness and one for safety (Alammar & Grootendorst, 2024, p. 383).
For an inferentialist, the procedure erases a distinction: that between taking something to be correct and its being correct. A reward model is a model of the approvals of a group of raters. Can a model of attitudes stand in for a model of norms? See inferentialism.