Methods that tune a model from comparisons between responses (“this one is better than that one”), either through a reward model and reinforcement learning (PPO) or directly (DPO).
The data take the form for this prompt, response A is better than response B
(Alammar & Grootendorst, 2024, pp. 380–381). People find comparing easier than scoring
(Christiano et al., 2017, sec. 1), and comparisons are modeled with Bradley–Terry: behind each choice
there is supposedly a latent score (see reward model).
PPO. The classic RLHF route trains a reward model r and then optimizes the
language model π with proximal policy optimization (Schulman et al., 2017), a
reinforcement learning algorithm tested on simulated robots and video games. The
objective is max E[r(x, y)] − β · KL(π ‖ π_ref): expected reward minus a penalty for drifting
from the reference model, so as not to exploit the judge's flaws (Ouyang et al., 2022, sec. 3.5). It
is costly and unstable: it requires sampling responses during training and keeping several
models at once (Alammar & Grootendorst, 2024, pp. 383–384).
DPO. Rafailov et al. (2023) noticed that the optimum of that objective has a
closed form. Substituting it into Bradley–Terry, the reward drops out and what remains is a
classification loss over pairs: raise the probability of the preferred response and lower
that of the rejected one, both relative to the reference model. Hence the subtitle: "your
language model is secretly a reward model." It is more stable and cheaper
(Alammar & Grootendorst, 2024, pp. 384–385). ORPO, in turn, merges supervised tuning and preferences into one
step (2024, p. 389).
Preferences need not be human: in constitutional AI another model supplies them
(Bai et al., 2022). And not all reinforcement is preference: the reasoning training of
DeepSeek-R1 used rule-based rewards (is the final answer correct?), and its
authors warn that a learned judge would be easier to exploit (Guo et al., 2025).
A strong assumption remains: Bradley–Terry puts everything on a single scale. Helpfulness,
truth, courtesy and style become commensurable, and the preferences of many people merge into
one; social choice theory has known since Arrow (1951) that aggregating preferences is not
neutral. What is lost when disagreement among raters is treated as noise?
Sources
- Alammar, J., & Grootendorst, M. (2024). Hands-on large language models: Language understanding and generation. O’Reilly Media. §
- Arrow, K. J. (1951). Social choice and individual values. Wiley. https://cowles.yale.edu/sites/default/files/2022-09/m12-all.pdf §
- Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., Chen, C., Olsson, C., Olah, C., Hernandez, D., Drain, D., Ganguli, D., Li, D., Tran-Johnson, E., Perez, E., . . . Kaplan, J. (2022). Constitutional AI: Harmlessness from AI feedback (arXiv:2212.08073). arXiv. https://doi.org/10.48550/arXiv.2212.08073 §
- Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., & Amodei, D. (2017). Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems (Vol. 30). Curran Associates. https://proceedings.neurips.cc/paper_files/paper/2017/hash/d5e2c0adad503c91f91df240d0cd4e49-Abstract.html §
- Guo, D., Yang, D., Zhang, H., Song, J., Wang, P., Zhu, Q., Xu, R., Zhang, R., Ma, S., Bi, X., Zhang, X., Yu, X., Wu, Y., Wu, Z. F., Gou, Z., Shao, Z., Li, Z., Gao, Z., Liu, A., . . . Zhang, Z. (2025). DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature, 645(8081), 633–638. https://doi.org/10.1038/s41586-025-09422-z §
- Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P. F., Leike, J., & Lowe, R. (2022). Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems (Vol. 35, pp. 27730–27744). Curran Associates. https://doi.org/10.52202/068431-2011 §
- Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., & Finn, C. (2023). Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems (Vol. 36, pp. 53728–53741). Curran Associates. https://doi.org/10.52202/075280-2338 §
- Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal policy optimization algorithms (arXiv:1707.06347). arXiv. https://doi.org/10.48550/arXiv.1707.06347 §
