T5 (Text-to-Text Transfer Transformer) is a transformer encoder–decoder, like the original, trained under a single rule: every task goes in as text and comes out as text. A Google team introduced it in 2019 as part of a systematic study of what works and what does not in transfer learning (Raffel et al., 2020).
Everything is text-to-text
To translate, T5 receives "translate English to German: That is good." and must produce "Das ist gut."; to judge whether a sentence is grammatical, it receives "cola sentence: …" and answers "acceptable" or "not acceptable"; to summarize, "summarize: …" (Raffel et al., 2020, pp. 3, 8). The name of the task is a natural-language prefix, and the model, objective, training procedure and decoding are the same for everything (2020, p. 2). Even sentence similarity, a number between 1 and 5, is rounded to steps of 0.2 and written as a string ("3.8"): regression becomes a 21-class classification (2020, p. 9). If the model answered "hamburger" to an inference question, the answer would count as wrong; the authors note that they never saw it happen (2020, p. 8).
How it works
The architecture is nearly that of the Transformer: in the base version, 12 blocks in the encoder and 12 in the decoder, each stack the size of BERT-base, about 220 million parameters in total (Raffel et al., 2020, p. 11; Alammar & Grootendorst, 2024, p. 128). The differences are details: a simplified normalization and, instead of sines and cosines, relative positions, one learned number per query–key distance (2020, pp. 4–5).
Pretraining is a variant of BERT's masked language modeling: spans of text are deleted and replaced by sentinel tokens; the encoder reads the perforated sentence and the decoder writes only what is missing. "Thank you for inviting me to your party last week" becomes "Thank you ⟨X⟩ me to your party ⟨Y⟩ week," and the target is "⟨X⟩ for inviting ⟨Y⟩ last ⟨Z⟩" (Raffel et al., 2020, p. 13; Alammar & Grootendorst, 2024, p. 129). The corpus is C4 (Colossal Clean Crawled Corpus), about 750 GB of English extracted from Common Crawl in April 2019 and filtered with simple rules: keep only lines ending in punctuation, and drop pages with code, with "lorem ipsum," or with any word from a list of obscene terms (Raffel et al., 2020, pp. 6–7).
A study more than a model
The paper is above all a battery of ablations: it compares architectures, objectives, corpora and transfer strategies, and finds that the encoder–decoder with a reconstruction objective beats single-stack variants such as a GPT-style decoder (Raffel et al., 2020, p. 19). With those conclusions and with scale (up to 11 billion parameters and about a trillion pretraining tokens), T5 reached the state of the art on 18 of 24 tasks and nearly matched human performance on SuperGLUE, 88.9 against 89.8 (2020, pp. 36–40).
Limits
The irony is that the heir of a translator did not reach the state of the art on any WMT translation task, partly because it was pretrained on English only (Raffel et al., 2020, p. 40). C4's filter has costs: the list, mostly of sexual terms, also removes texts from LGBTQ communities that use those words in a reclaiming sense (Bender et al., 2021, p. 614). And the 11B version weighs tens of gigabytes. Its liveliest lineage is Flan-T5, fine-tuned on more than a thousand tasks phrased as instructions (Alammar & Grootendorst, 2024, p. 130). The form also survived by the reverse route: T5Gemma takes an already pretrained decoder-only model (Gemma 2) and adapts it into an encoder–decoder; at a similar inference budget it performs about the same in pretraining and markedly better after fine-tuning (Zhang et al., 2025). Its successor, T5Gemma 2, starts from Gemma 3 and takes images as input (Zhang & Hora, 2025).