RLHF was born for a problem in simulated robotics: how do you teach a behavior for which nobody can write a reward function? Christiano et al. (2017) showed a person two clips of 1 to 2 seconds, asked which one they preferred, and trained a reward predictor on those comparisons; with 900 queries, in under an hour of human time, a simulated robot learned to do backflips (2017, secs. 2.2, 3.2).
Applied to language models, the canonical procedure is that of InstructGPT, the forerunner of ChatGPT (Ouyang et al., 2022, sec. 3.1):
- Demonstrations: people write ideal answers, and the model undergoes instruction tuning.
- Comparisons: people rank several model outputs; these rankings train a reward model.
- Reinforcement: the model is optimized with PPO to maximize that reward, with a penalty that keeps it from drifting too far from the step-1 model and "gaming" the judge (2022, sec. 3.5).
The result drew attention: outputs of a 1.3-billion-parameter InstructGPT were preferred to those of the 175-billion-parameter GPT-3 (Ouyang et al., 2022). Later variants such as DPO dispense with a separate reward model (Alammar & Grootendorst, 2024, pp. 383–385), and Constitutional AI replaces part of the human judgment with the judgment of another model.
The fine print matters. The authors themselves ask whom the model is being aligned to: about 40 hired contractors, mostly in the United States and Southeast Asia, who agreed with each other roughly 73% of the time, following instructions written by the company (Ouyang et al., 2022, sec. 5.2). And when launching ChatGPT, OpenAI acknowledged that during reinforcement-learning training "there's currently no source of truth" (OpenAI, 2022). The p(doom) song sums it up in one line: "RLHF goes askew" (osmarks, n.d.). For Millière (2025), RLHF instills shallow behavioral dispositions rather than a capacity for normative deliberation, hence its fragility in the face of a jailbreak.