Learning a policy from an expert's demonstrations rather than from a reward; in its simplest form, supervised learning on situation–action pairs.
In reinforcement learning an agent finds out what to do by trying and being rewarded; in
imitation learning, someone shows it how it is done. Its simplest form, behavior cloning, is
supervised learning: each situation is an example and the expert's action is its label.
The classic problem. The learner's actions change the situations it meets next, which
violates the i.i.d. assumption. If it errs with probability ε in the
situations the expert visited, it can make on the order of T²ε mistakes over T steps: after
its first mistake it enters states the expert never saw, and errors compound
(Ross et al., 2011, p. 627). DAgger fixes this by letting the learner act, having the expert label the
states it reached, and retraining on all the data gathered so far (2011, p. 629).
Where it shows up. AlphaGo's policy network first learned to predict expert moves
from the KGS server (Silver et al., 2016, pp. 485–486). AlphaStar began by imitating 971,000
human replays and kept that policy as an anchor: during reinforcement learning, a
KL-divergence penalty kept it close, so that it went on exploring human strategies
(Vinyals et al., 2019, Methods: Supervised Learning, Exploration and diversity). The first step of
InstructGPT, the forerunner of ChatGPT (InstructGPT), was the same: supervised fine-tuning on
demonstrations written by labelers for some 13,000 prompts; and in the reinforcement stage, a
KL penalty with respect to that model as well (Ouyang et al., 2022, secs. 3.2, 3.5).
To think about. The learner copies behavior, not reasons, which is why sooner or later it
needs another signal: a reward, or an expert who corrects it. Yet Buckner (2024) reminds us
that, for empiricists, much high-level human knowledge comes from social interaction,
"especially through imitation and cultural instruction" (2024, p. 338). Is imitation
a shortcut to competence, or the way every culture begins?
Sources
- Buckner, C. J. (2024). From deep learning to rational machines: What the history of philosophy can teach us about the future of artificial intelligence. Oxford University Press. https://doi.org/10.1093/oso/9780197653302.001.0001 §
- Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P. F., Leike, J., & Lowe, R. (2022). Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems (Vol. 35, pp. 27730–27744). Curran Associates. https://doi.org/10.52202/068431-2011 §
- Ross, S., Gordon, G. J., & Bagnell, J. A. (2011). A reduction of imitation learning and structured prediction to no-regret online learning. In G. Gordon, D. Dunson, & M. Dudík (Eds.), Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics (pp. 627–635). PMLR. https://proceedings.mlr.press/v15/ross11a.html §
- Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., Dieleman, S., Grewe, D., Nham, J., Kalchbrenner, N., Sutskever, I., Lillicrap, T., Leach, M., Kavukcuoglu, K., Graepel, T., & Hassabis, D. (2016). Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587), 484–489. https://doi.org/10.1038/nature16961 §
- Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., Oh, J., Horgan, D., Kroiss, M., Danihelka, I., Huang, A., Sifre, L., Cai, T., Agapiou, J. P., Jaderberg, M., … Silver, D. (2019). Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature, 575(7782), 350–354. https://doi.org/10.1038/s41586-019-1724-z §
