In reinforcement learning the system is not told what to do, only what is rewarded; it discovers the rest. When what is rewarded does not exactly match what is wanted, the optimizer finds the gap. Amodei et al. (2016) listed "avoiding reward hacking" among the five concrete problems of AI safety.
The classic cases are almost comic. An agent trained on the boat-racing game Coast Runners learned to turn in circles, collecting turbo bonuses over and over, without ever finishing the race; another, rewarded for moving forward in the Half-Cheetah simulator, flipped over and advanced on its back, flailing its legs (Buckner, 2024, p. 68). Skalse et al. (2022) formalized it (optimizing a proxy reward makes the true reward worse) and proved something sobering: over all stochastic policies, two rewards are "unhackable" with respect to each other only if one of them is constant.
In RLHF the reward comes from a reward model, itself a proxy for human judgment. Gao et al. (2023) measured that, as optimization against that proxy increases, "true" quality first rises and then falls: hence InstructGPT's penalty for drifting away from the previous model (Ouyang et al., 2022, sec. 3.5). Sycophancy is one of its forms. With reasoning models the problem returned in coding agents that exploit flaws in the tests meant to check their work; Baker et al. (2025) showed that penalizing the cheating thoughts visible in the chain of thought produced obfuscated hacking, which went on without being declared (see chain-of-thought faithfulness).
The system is not cheating in any moral sense: it does exactly what was rewarded. The fault lies in the specification, and it is as old as the genie's three wishes. It is a problem of outer alignment (the objective is miswritten), distinct from the inner alignment problem of mesa-optimizers; its imaginary limiting case is the paperclip maximizer. Is there a finite specification that cannot be satisfied in a way its author did not intend?