In the scheme of Brown et al. (2020), zero-shot is the far end of the in-context learning spectrum: the model receives only an instruction (for example, "Translate English to French:") and the case to solve, with no demonstrations (2020, sec. 2). It is the most convenient setting and the closest to how a task is assigned to a person, but also the hardest: sometimes even a human would not understand the expected format without an example. The authors give one: "make a table of world records for the 200m dash" leaves open which columns to include.
The term is misleading, and the authors themselves say so: it is called "zero-shot" because there are no weight updates, but the model is not learning "from zero examples"; it may have seen thousands of similar tasks during pretraining (Brown et al., 2020, sec. 1). The expression also has a prior history in machine learning: classifying into categories never seen during training. This is the sense in which CLIP uses it, classifying an image by comparing its embedding with those of descriptions of the candidate classes (Alammar & Grootendorst, 2024, pp. 123, 265).
Two findings show how much zero-shot performance depends on apparently minor details. Instruction tuning improves it greatly: FLAN outperformed zero-shot GPT-3 on 20 of 25 datasets (Wei, Bosma, et al., 2022). And a single phrase can change everything: adding "Let's think step by step" took a GPT-3 model from 17.7% to 78.7% on arithmetic word problems (Kojima et al., 2022; Buckner, 2024, p. 280). If a magic formula quadruples performance, what was the test measuring without it: the model's ability, or our skill at asking for it?