The degree to which the chain of thought a model writes reflects the process that actually produced its answer, rather than an after-the-fact rationalization.
A chain of thought improves answers, but does it explain them? An explanation is
faithful if it describes the factors that actually caused the answer. The question matters
because reasoning models display their steps, and it is tempting to read
them as a window onto the process.
The evidence counsels caution. Turpin et al. (2023) biased the examples in a prompt
so that the right answer was always "(A)": models leaned toward (A) even when it was wrong,
wrote chains justifying it, and systematically failed to mention the pattern; accuracy fell
by as much as 36%. Lanham et al. (2023) intervened on the chains (truncating them, inserting
mistakes) and found that some models relied heavily on what was written while others nearly
ignored it; on most tasks, larger models reasoned less faithfully. Chen et al. (2025)
slipped hints into reasoning models' prompts: when the models used them, they
acknowledged them in the chain often less than 20% of the time.
There is also a design risk. Baker et al. (2025) showed that having another model
read the chain detects reward hacking, but that if cheating thoughts are penalized in
training, the model learns to hide them and keeps cheating. Hence a broad group of
researchers (Korbak et al., 2025) described monitorability as a "new and fragile"
safety opportunity.
Philosophers have an uncomfortable precedent: Nisbett & Wilson (1977) showed that people
confidently explain the causes of their own behavior and are often wrong, relying on
plausible theories rather than introspection. Our own reports are not faithful by default
either. And one distinction remains: a chain can be causally useful, as a scratchpad that
gives the model more computation, without being a faithful
description of that computation. Is the chain a report, a tool, or a performance?
Sources
- Baker, B., Huizinga, J., Gao, L., Dou, Z., Guan, M. Y., Madry, A., Zaremba, W., Pachocki, J., & Farhi, D. (2025). Monitoring reasoning models for misbehavior and the risks of promoting obfuscation (arXiv:2503.11926). arXiv. https://doi.org/10.48550/arXiv.2503.11926 §
- Chen, Y., Benton, J., Radhakrishnan, A., Uesato, J., Denison, C., Schulman, J., Somani, A., Hase, P., Wagner, M., Roger, F., Mikulik, V., Bowman, S. R., Leike, J., Kaplan, J., & Perez, E. (2025). Reasoning models don't always say what they think (arXiv:2505.05410). arXiv. https://doi.org/10.48550/arXiv.2505.05410 §
- Korbak, T., Balesni, M., Barnes, E., Bengio, Y., Benton, J., Bloom, J., Chen, M., Cooney, A., Dafoe, A., Dragan, A., Emmons, S., Evans, O., Farhi, D., Greenblatt, R., Hendrycks, D., Hobbhahn, M., Hubinger, E., Irving, G., Jenner, E., . . . Mikulik, V. (2025). Chain of thought monitorability: A new and fragile opportunity for AI safety (arXiv:2507.11473). arXiv. https://doi.org/10.48550/arXiv.2507.11473 §
- Lanham, T., Chen, A., Radhakrishnan, A., Steiner, B., Denison, C., Hernandez, D., Li, D., Durmus, E., Hubinger, E., Kernion, J., Lukošiūtė, K., Nguyen, K., Cheng, N., Joseph, N., Schiefer, N., Rausch, O., Larson, R., McCandlish, S., Kundu, S., . . . Perez, E. (2023). Measuring faithfulness in chain-of-thought reasoning (arXiv:2307.13702). arXiv. https://doi.org/10.48550/arXiv.2307.13702 §
- Nisbett, R. E., & Wilson, T. D. (1977). Telling more than we can know: Verbal reports on mental processes. Psychological Review, 84(3), 231–259. https://doi.org/10.1037/0033-295X.84.3.231 §
- Turpin, M., Michael, J., Perez, E., & Bowman, S. R. (2023). Language models don't always say what they think: Unfaithful explanations in chain-of-thought prompting. In Advances in Neural Information Processing Systems (Vol. 36, pp. 74952–74965). Curran Associates. https://doi.org/10.52202/075280-3275 §
