Constitutional AI, proposed by Bai et al. (2022) at Anthropic, aims to make a model harmless without human labels about what is harmful. Human oversight is reduced to a list of natural-language principles (on the order of ten) plus a few examples (Bai et al., 2022, secs. 1.1–1.2). The process has two stages:
- Supervised: critique → revision. A helpful but unfiltered assistant responds to harmful requests; it is then asked to critique its own response according to a randomly drawn principle and to rewrite it. The revised responses are used for fine-tuning.
- Reinforcement learning from AI feedback (RLAIF). The model generates pairs of responses and another model decides which one better satisfies a principle. These artificial preferences train a preference model (a reward model), against which the assistant is optimized as in RLHF.
Two clarifications. First, humans are not eliminated: preferences about helpfulness are still human; only those about harmlessness come from the AI (2022, sec. 1.2). Second, an explicit goal was an assistant that is "harmless but non-evasive," one that explains why it will not help instead of simply refusing, a flaw the authors attributed to human raters rewarding evasiveness.
The authors justify the name with an observation from political philosophy: when deploying a general system, one cannot avoid choosing principles to govern it, even if they remain hidden or implicit (2022, sec. 1). But there is a question from philosophy of language underneath: principles do not apply themselves; another language model interprets them. What guarantees that this interpretation matches what the drafters meant? It is the rule-following problem, now with a machine on each side. For Millière (2025), this kind of alignment remains shallow: it trains dispositions, not the capacity to resolve conflicts between norms.