GPT-1 is a transformer decoder that first learns to predict the next word in thousands of books and is then fine-tuned, with minimal changes, for comprehension tasks. OpenAI released it in June 2018 (Radford et al., 2018); it had 117 million parameters (Radford et al., 2019, sec. 3; Alammar & Grootendorst, 2024, p. 20).
The G, the P and the T
- G for generative. Pretraining is a classic language model: maximize the probability of each token given the preceding ones (Radford et al., 2018, sec. 3.1). Such a model can generate text, although the paper does not use it for that.
- P for pretrained. There are two stages: unlabeled pretraining and supervised fine-tuning for each task, keeping language modeling as an auxiliary objective (2018, secs. 3.1–3.2). The underlying argument: unlabeled text is abundant and labeled data is scarce (2018, sec. 1).
- T for transformer. It uses only the right half of the Transformer, without cross-attention: 12 blocks with masked self-attention, 768-dimensional vectors, 12 heads, a 512-token context and a BPE vocabulary of 40,000 merges (2018, sec. 4.1). It was chosen because it offers a "more structured memory" than LSTMs for long-range dependencies (2018, sec. 1).
The acronym does not appear in the paper, which calls its model "Finetuned Transformer LM"; the expansion Generative Pre-trained Transformer appears, for instance, in the BERT paper (Devlin et al., 2019, p. 4171), and the "-1" was added when successors arrived (Alammar & Grootendorst, 2024, p. 20).
How it adapts to each task
The practical novelty is the input transformations. Instead of designing an architecture per
task, structured inputs are flattened into a single sequence with delimiters: for inference,
premise and hypothesis separated by a $ symbol; for similarity, both sentences in both
orders; for multiple choice, context, question and each candidate answer separately
(Radford et al., 2018, sec. 3.3). The only new parameters in fine-tuning are a linear output layer
and the delimiter embeddings (2018, sec. 3.2).
What it achieved
It improved the state of the art on 9 of 12 datasets, with gains of 8.9 points on Story Cloze and 5.7 on RACE (middle and high school reading comprehension exams); on GLUE it went from 68.9 to 72.8 (Radford et al., 2018, sec. 4.2). The ablations speak clearly: without pretraining, the average drops by 14.8%; with an LSTM instead of the transformer, by 5.6 points (2018, sec. 5).
More important in the long run was a side observation. With no fine-tuning at all, the pretrained model already solved part of the tasks through simple heuristics: for sentiment, the word very was appended to the review and the probabilities of positive and negative were compared. This zero-shot performance grew with pretraining, and the authors conjectured that the generative model learns to perform many tasks in order to improve its language modeling (2018, sec. 5). That conjecture is the seed of GPT-2.
Limits
It only looks left. BERT, which chose for its base version the same size as GPT-1 so the two could be compared, beat it precisely through bidirectionality (Devlin et al., 2019, pp. 4171, 4173). It still needs labeled data, and on small datasets such as RTE it stayed below the state of the art (Radford et al., 2018, sec. 4.2). Its corpus was not innocent either: a later audit of BooksCorpus found thousands of duplicated books, skewed genre representation and likely copyright violations (Bandy & Vincent, 2021). Some summaries add Common Crawl (Alammar & Grootendorst, 2024, p. 20), but the paper describes only BooksCorpus, about 800 million words (Radford et al., 2018, sec. 4.1; Devlin et al., 2019, p. 4184).