GPT-3 is a 175-billion-parameter autoregressive generation language model, ten times larger than any previous dense model, released by OpenAI in 2020 (Brown et al., 2020, sec. 1). Its fame does not come from its architecture, which is essentially that of GPT-2, but from a behavior: solving new tasks from a handful of examples written into the prompt, without any fine-tuning.
How it works
It is a stack of 96 decoder blocks with a causal mask, 12,288-dimensional vectors and 96 attention heads; a context window of 2048 tokens; and layers that alternate dense and locally banded sparse attention (Brown et al., 2020, sec. 2.1; Alammar & Grootendorst, 2024, p. 96). It was trained on 300 billion tokens from a mixture in which the sources deemed higher quality (WebText2, Books1, Wikipedia) were seen two or three times, and the filtered Common Crawl and Books2 less than once (2020, sec. 2.2). The cost was about 3.14 × 10²³ FLOPs (2020, app. D).
The size was not a whim. Months earlier, the scaling laws of Kaplan et al. (2020) had shown that loss falls as a power law in parameters, data and compute, and that a budget is best spent mostly on bigger models (2020, sec. 1.1). GPT-3 followed that recipe: many parameters, relatively few tokens (Brown et al., 2020, fig. 2.2).
Learning without touching the weights
The paper defines three settings (2020, sec. 2). In zero-shot the model receives only an instruction; in one-shot, an instruction and one example; in few-shot, as many examples as fit in the window (typically 10 to 100). In none of them are gradients updated: the model simply continues the text. The authors call this in-context learning, and Alammar & Grootendorst (2024) present it as the idea of showing the task instead of describing it (2024, pp. 180–181).
Some results: 71.2% on TriviaQA without consulting documents, 86.4% on LAMBADA when the task is framed as a cloze test with examples, 100% on two-digit addition (Brown et al., 2020, secs. 3.1, 3.2, 3.9.1). In an experiment with about 80 people per model, accuracy at telling GPT-3's news articles from real ones was ≈ 52%, barely above chance (2020, sec. 3.9.4). The most interesting finding concerns shape: the gap between zero, one and few examples widens with model size, as if larger models were better "meta-learners" (2020, sec. 1).
What changed
With GPT-3 the prompt became a way of programming. That brought "prompt engineering" and its fragility: tiny changes in punctuation or wording alter performance (Buckner, 2024, pp. 278–279). It also made clear that a base model is not an assistant: it predicts continuations rather than following instructions, which is why its descendants went through instruction tuning and RLHF on the way to ChatGPT (InstructGPT) (Alammar & Grootendorst, 2024, pp. 26, 88).
Limits
The authors list their own (Brown et al., 2020, sec. 5): incoherence over long passages, near-chance results on comparison tasks such as WiC and ANLI, trouble with everyday physics, poor sample efficiency in pretraining, and the biases of the data. A filtering bug left some data contamination with benchmarks that could not be fixed because retraining was too expensive (2020, secs. 2.2, 4). And since weights and data stayed with OpenAI, studying it reproducibly was far harder than studying BERT (Buckner, 2024, pp. 279–280).