Transfer learning starts from a simple idea: what is learned on one problem helps with another. In its most elementary form, the intermediate representations of an already trained model are reused as the starting point of a new one (what Keras calls feature extraction) (Chollet, 2021, sec. 7.2.2). The first layers of a vision network detect edges and textures useful for almost any image; the first layers of a language model, regularities of the language useful for almost any text.
Between 2018 and 2020 this idea reorganized language processing: one expensive round of pretraining, then many cheap rounds of fine-tuning, one per task (Alammar & Grootendorst, 2024, p. 19). BERT, GPT-1 and T5 follow the recipe; the T5 paper puts the word in its title and systematically compares its variants (Raffel et al., 2020). Kaplan et al. (2020) also observed that when a model is evaluated on text from a different distribution, performance drops by a roughly constant amount but improves at the same pace as on the training distribution (2020, sec. 1.1). With GPT-3, transfer moves into the prompt: the model adapts to new tasks without changing its weights, through in-context learning (Brown et al., 2020, sec. 2).
Philosophically, transfer is the everyday face of generalization, and therefore of induction. Chollet (2019) distinguishes local generalization (new cases from a known distribution), broad generalization (new tasks within a category) and extreme generalization (tasks that share only abstract features with what is known) (2019, sec. I.3.2). Which rung is reached when a model goes from "predict the next word" to classifying reviews or translating? The answer depends on how much of the new task was already, implicitly, in the corpus.