GPT-2 is GPT-1 at ten times the scale with a new corpus: a transformer decoder with 1.5 billion parameters trained to predict the next token on web pages. OpenAI announced it in February 2019, and it became famous both for what it did without task-specific training and for what its makers at first refused to publish.
Bigger, nearly the same
The paper trained four sizes, from 117 to 1,542 million parameters; the largest, the only one the paper calls GPT-2, has 48 layers and 1,600-dimensional vectors (Radford et al., 2019, secs. 2.3, 3; Alammar & Grootendorst, 2024, p. 20). The architectural changes are minor: layer normalization moves to the input of each sub-block, the context doubles to 1,024 tokens, and the vocabulary, a byte-level BPE, has 50,257 entries, so any Unicode string can be represented (Radford et al., 2019, secs. 2.2–2.3; Alammar & Grootendorst, 2024, p. 49).
A corpus chosen by votes
OpenAI wanted varied text, but Common Crawl was full of unintelligible documents, and manually filtering a web scrape would have been exceedingly expensive. The solution was to delegate curation to Reddit: they took the outbound links posted there that had received at least 3 karma, a "heuristic indicator" that other users had found them interesting, educational, "or just funny" (Radford et al., 2019, sec. 2.1). Since karma is a net vote balance, that means roughly three upvotes (Mollick, 2026). From those 45 million links came WebText: slightly over 8 million documents and 40 GB of text, with no links created after December 2017 and no Wikipedia, which was removed to avoid contaminating evaluations (Radford et al., 2019, sec. 2.1). Upvotes had been invented to rank links, not to teach a machine to write (Mollick, 2026).
Tasks without training
The paper's thesis is in its title: language models are unsupervised multitask learners. A task can be written down as text, and a model that predicts the web well should learn to do the tasks the web displays, because that helps it predict (Radford et al., 2019, sec. 2). WebText does in fact contain spontaneous translations between English and French (2019, Table 1).
With no fine-tuning at all (zero-shot), GPT-2 improved the state of the art on 7 of 8 language modeling datasets (Radford et al., 2019, sec. 3.1). To summarize, it was enough to append "TL;DR:" to an article; without that hint, performance dropped by 6.4 points (2019, sec. 3.6). To translate, it was given example pairs in the format "english sentence = french sentence" (2019, sec. 3.7): a preview of the prompt and of the few-shot learning of GPT-3.
"Too dangerous"
The February announcement published the paper and the small model but withheld the larger ones for fear of malicious uses: fake news, impersonation, automated abusive content. A staged release followed: 355 million parameters in May, 774 million in August, and the full model in November 2019. The final report found no evidence of direct misuse in public forums, though it did find discussions of how to try (Solaiman et al., 2019). The press summed it up as the model "too dangerous to release."
Limits
Without fine-tuning, results were promising but modest: on open questions it was right 4.1% of the time (Radford et al., 2019, sec. 3.8), and its summaries barely beat picking three sentences at random (2019, sec. 3.6). It memorized heavily repeated texts, such as the Gettysburg Address (2019, sec. 8.2). And its prose was fluent and often false: in the most quoted sample (the best of ten), the Andean unicorns had four horns (2019, Table 13).