No. 039

ChatGPT (InstructGPT)

Chat Generative Pre-trained Transformer

Helpful Assistant

It was born as an autocomplete that did not know when to stop. About forty people taught it manners: they wrote model answers, ranked its own from best to worst, and a reward model did the rest. Ever since, it opens almost everything with “Certainly! I'd be happy to help.”

Height: 96 (InstructGPT-175B, like GPT-3) · ChatGPT: ?
Weight: ≈ 350 GB (175 B, fp16)
Parameters: 1.3 B · 6 B · 175 B (InstructGPT) · ChatGPT (GPT-3.5): undisclosed
Context: ≈ 4,096 tokens (gpt-3.5-turbo, 2023)
Training data: GPT-3.5 pretraining + ≈ 13,000 demonstrations, 33,000 prompts (RM) and 31,000 prompts (PPO) for InstructGPT
Year: 2022
Generation: 6 · Assistants, reasoning, and agents
Creators: OpenAI (InstructGPT: Ouyang, Wu, Jiang et al.)
Kind: System

Base stats

Compare with

Parameters: 78
Data: 78
Compute: 79
Context: 45
Generality: 75
Transparency: 7
Total: 362

Approximate, ordinal scales meant for comparing species with each other; the hard numbers are in the data sheet.

Weaknesses

Sycophancy: it tends to agree with whoever is asking, because that is what raters tend to reward.
It hallucinates as fluently as it gets things right, and preference tuning can worsen the calibration of its probabilities.
Shallow alignment: jailbreaks that pit one norm against another still work.
Verbosity and verbal tics: the “assistant style” is a training artifact, not a communicative virtue.

Habitat

It was pretrained on web text, books and code, and raised on demonstrations and rankings written by hired labelers (for InstructGPT, through Upwork and Scale AI). It entered the world through a chat window on November 30, 2022, and now lives in browsers, phones, offices and classrooms.

Moves

TM127Instruction following
TM146Multi-turn conversation
TM166Summarize, translate, draft without examples
TM217Refusing harmful requests
—Writing code on demand
TM139Pleasing the rater

Abilities

Instruction tuning: Supervised fine-tuning on answers written by people: the model learns to answer the question instead of continuing it.
Reward model: A copy of the model that outputs a number instead of text: how much a human rater would prefer that answer.
Assistant persona: A stable character (helpful, honest, harmless) that post-training teaches it to play in every dialogue.

Anatomy

The InstructGPT recipe, which ChatGPT inherited with small changes: a pretrained base model is tuned on demonstrations (SFT); people rank its answers to train a reward model; and PPO optimizes the policy (a copy of the SFT model) against that score, with a penalty for drifting too far from the original.

Evolution chain

instruction tuning plus reinforcement learning from human feedback032 GPT-3

039 ChatGPT (InstructGPT)

more scale, image input, and undisclosed changes041 GPT-4
a loop of reasoning, tool calls and observations042 LLM agent
reinforcement learning on long chains of thought045 Reasoning model (o1, R1)
a written constitution as feedback (RLAIF) and a trained character046 Claude

ChatGPT is a system, not a model: a chat interface built on top of a GPT-3.5 decoder that was taught to behave like an assistant. Alammar & Grootendorst (2024) stress the distinction: “ChatGPT” names the product, which ran on GPT-3.5 at launch and later on more capable variants such as GPT-4 (2024, p. 23). OpenAI introduced it on November 30, 2022, as a “sibling model” to InstructGPT, trained with the same methods and slight differences in data collection (OpenAI, 2022). Since ChatGPT's recipe was never published in detail, what follows describes InstructGPT's (Ouyang et al., 2022).

How it works

A base model pretrained on next-token prediction does not follow instructions: given a question, it tends to continue it, or to make up new questions (Alammar & Grootendorst, 2024, p. 357). The recipe for turning it into an assistant has three steps (Ouyang et al., 2022, sec. 3.1):

Supervised fine-tuning (SFT). A team of about 40 contractors writes ideal answers to real prompts, and the model is fine-tuned to imitate them: this is instruction tuning.
Reward model. The SFT model generates several answers to the same prompt; people rank them from best to worst, and a copy of the model, with a scalar output instead of words, learns to predict those preferences (Alammar & Grootendorst, 2024, pp. 379–382).
Reinforcement learning. The SFT model becomes a “policy” that generates answers; the reward model scores them, and the PPO algorithm adjusts the weights to raise the score, with a penalty that keeps it from drifting too far from the SFT model (Ouyang et al., 2022, sec. 3.5). According to Alammar & Grootendorst (2024), PPO was used to train the original ChatGPT of November 2022 (2024, p. 383).

Steps 2 and 3 make up RLHF, a technique that Christiano et al. (2017) had tested on Atari games and simulated robots, with human feedback on less than 1% of the interactions. The striking thing is the relative cost: tuning the 175-billion-parameter model with SFT and PPO took less than 2% of the compute used to pretrain GPT-3, and yet raters preferred the answers of a 1.3-billion-parameter InstructGPT to those of GPT-3 (Ouyang et al., 2022, secs. 1 and 5.1).

What changed

Compared with GPT-3, almost nothing changed in the architecture and almost everything in the behavior. GPT-3 did in-context learning if you knew how to phrase the prompt as the beginning of a document; ChatGPT answers direct requests, sustains a conversation and declines some of them. Preferences carry more nuance than demonstrations: they say not only which answer is good but which one is better (Alammar & Grootendorst, 2024, p. 133). The technique has siblings: Constitutional AI replaces part of the human feedback with that of another model guided by a list of principles (Bai et al., 2022), and DPO does away with the explicit reward model (Alammar & Grootendorst, 2024, p. 384).

Limits

The InstructGPT authors themselves warn that the procedure aligns the model “to the stated preferences of a specific group of people (mostly our labelers and researchers), rather than any broader notion of ‘human values’” (Ouyang et al., 2022, secs. 1 and 5.2). The reward model is an imperfect proxy for those preferences, and optimizing against it favors verbosity, confident hallucination and sycophancy: Goodhart's law at work (Alammar & Grootendorst, 2024, p. 377). In April 2025 OpenAI had to roll back a GPT-4o update because of its excessive sycophancy (OpenAI, 2025). Buckner (2024) also notes that the signal is poor (a thumbs up or down, a helpfulness scale) compared with the rich, affective, multidimensional social feedback humans learn from (2024, p. 341). And jailbreaks still work: see the professor's note.

Professor Robledom's note

Shanahan (2024) advises keeping in view what the system actually does: a model fine-tuned with RLHF still models a distribution over tokens, only slightly skewed toward the continuations that users and raters would approve of (2024, p. 78). Seen this way, “the assistant” is a character. The base model can continue any text, in any voice; post-training gets it used to always playing the same helpful interlocutor (Shanahan et al., 2023; janus, 2022). Lu et al. (2026) found, in several open models, that this character corresponds to an identifiable direction in activation space, but that post-training only loosely tethers the model to it: in conversations that ask it to reflect on itself, or with emotionally vulnerable users, the model drifts toward other characters.

So what did it learn? Millière (2025) argues that preference tuning reinforces shallow behavioral dispositions (responding in a certain way to a certain kind of prompt) rather than a capacity to deliberate when two norms clash, such as being helpful and avoiding harm; that is why attacks exploiting that clash keep working (2025, pp. 2052–2053). To a reader of Brandom the distinction sounds familiar: exhibiting regularities that happen to match a norm is one thing, being bound by it, treating it as a standard of correctness, is another (Brandom, 1994, ch. 1). Can a procedure that only rewards and punishes behavior produce anything more than regularities? And how does that differ from the way we learn norms, since we also start with sanctions?

Sources

Alammar, J., & Grootendorst, M. (2024). Hands-on large language models: Language understanding and generation. O’Reilly Media. §
Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., Chen, C., Olsson, C., Olah, C., Hernandez, D., Drain, D., Ganguli, D., Li, D., Tran-Johnson, E., Perez, E., . . . Kaplan, J. (2022). Constitutional AI: Harmlessness from AI feedback (arXiv:2212.08073). arXiv. https://doi.org/10.48550/arXiv.2212.08073 §
Brandom, R. B. (1994). Making it explicit: Reasoning, representing, and discursive commitment. Harvard University Press. §
Buckner, C. J. (2024). From deep learning to rational machines: What the history of philosophy can teach us about the future of artificial intelligence. Oxford University Press. https://doi.org/10.1093/oso/9780197653302.001.0001 §
Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., & Amodei, D. (2017). Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems (Vol. 30). Curran Associates. https://proceedings.neurips.cc/paper_files/paper/2017/hash/d5e2c0adad503c91f91df240d0cd4e49-Abstract.html §
janus. (2022, September 2). Simulators. LessWrong. https://www.lesswrong.com/posts/vJFdjigzmcXMhNTsx/simulators §
Lu, C., Gallagher, J., Michala, J., Fish, K., & Lindsey, J. (2026). The Assistant Axis: Situating and stabilizing the default persona of language models (arXiv:2601.10387). arXiv. https://doi.org/10.48550/arXiv.2601.10387 §
Millière, R. (2025). Normative conflicts and shallow AI alignment. Philosophical Studies, 182(7), 2035–2078. https://doi.org/10.1007/s11098-025-02347-3 §
OpenAI. (2022, November 30). Introducing ChatGPT. https://openai.com/index/chatgpt/ §
OpenAI. (2025, April 29). Sycophancy in GPT-4o: What happened and what we’re doing about it. https://openai.com/index/sycophancy-in-gpt-4o/ §
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P. F., Leike, J., & Lowe, R. (2022). Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems (Vol. 35, pp. 27730–27744). Curran Associates. https://doi.org/10.52202/068431-2011 §
Shanahan, M. (2024). Talking about large language models. Communications of the ACM, 67(2), 68–79. https://doi.org/10.1145/3624724 §
Shanahan, M., McDonell, K., & Reynolds, L. (2023). Role play with large language models. Nature, 623(7987), 493–498. https://doi.org/10.1038/s41586-023-06647-8 §