A reasoning model is a language model trained to write a long chain of thought before answering, and to write it so that it leads to correct answers. OpenAI presented o1 on September 12, 2024: according to the company, a large-scale reinforcement learning algorithm teaches the model “how to think productively using its chain of thought,” and performance improves both with more reinforcement during training and with more thinking time at use (OpenAI, 2024). DeepSeek released R1 in January 2025, with the weights and a description of the method that later appeared in Nature (Guo et al., 2025).
From trick to training
The chain of thought began as a prompt trick: showing a few examples worked out step by step improved large models on arithmetic and logic problems (Wei, Wang, et al., 2022), and simply asking “Let's think step by step” was enough (Alammar & Grootendorst, 2024, pp. 185–187). Alammar & Grootendorst (2024) give a computational explanation: each additional token lets the model spread more computation over the problem instead of solving it in a few steps (2024, pp. 185–186). Buckner (2024) saw in these prompts the kind of scaffolding a teacher offers a child who is learning increasingly complex problems (2024, p. 340). Reasoning models internalize the trick: instead of asking for it, it is trained in.
How it works
R1-Zero starts from the DeepSeek-V3 base model and skips supervised fine-tuning. It is trained
with reinforcement (the GRPO algorithm) on math, code and science problems, and the reward
depends only on whether the final answer matches the solution and whether the reasoning sits
between <think> tags. The authors avoided neural reward models because, in
large-scale reinforcement, the model learns to game them. On AIME 2024, a math olympiad
qualifier, average accuracy rose from 15.6% to 77.9% during training; answers grew longer on
their own, and at some point an “aha moment” appeared, a sudden rise in the use of the word
wait in reflections (Guo et al., 2025). R1-Zero mixed English and Chinese and was hard to read;
R1 adds initial examples with readable chains, more stages of reinforcement and tuning, and
preference rewards for general use. Its reasoning was later distilled into
models from 1.5 B to 70 B.
What changed
In ChatGPT (InstructGPT), the reward model judged whole answers by human preferences; here the reward comes from a verifier, a cheap, objective and unlimited signal in math and code. A new axis of scaling also appears, test-time compute: o1 solved on average 74% of AIME 2024 problems with a single sample and 83% with consensus among 64 (OpenAI, 2024). OpenAI decided not to show the raw chains; R1 displays them. When R1 came out, free and with open weights, Nvidia's stock plunged (see NVDA to the moon).
The p(doom) song ends by asking “What did Ilya see?”. In November 2023 OpenAI's board, with Ilya Sutskever's vote, removed Sam Altman for not being “consistently candid” and reinstated him five days later (see the board crisis); stories circulated about a secret math-reasoning project, Q*. The song's annotator conjectures that what Ilya saw was the reinforcement training of chains of thought that led to o1 (osmarks, n.d.). It is a conjecture, not a fact.
Limits
The advantage is concentrated in verifiable domains: its own authors acknowledge that R1-Zero, trained only with rule-based rewards, performed poorly at writing and open-ended questions (Guo et al., 2025). Reinforcement against a verifier inherits the risk of all reinforcement, reward hacking: optimizing the signal instead of the task (Buckner, 2024, p. 68). And the chain of thought is a new attack surface: Millière (2025) shows cases in which R1 mentions safety in its trace and then produces the harmful content anyway, or leaks it in the trace while refusing it in the answer (2025, pp. 2054–2055).