A model is autoregressive when it consumes its own earlier predictions to make later ones. A generative LLM does not produce its answer in one go: on each pass it computes a distribution over the next token, picks one, appends it to the text and starts again; a program around the network runs this loop until it reaches an end-of-text token or a limit (Alammar & Grootendorst, 2024, pp. 74–76). That is why text generators are often called “autoregressive models”, in contrast with encoders such as BERT (Alammar & Grootendorst, 2024, p. 76). The term comes from time-series statistics, where a value is “regressed” on its own past values.
Tiny example. Input: “The capital of France is”. Step 1: the model proposes Paris. New input: “The capital of France is Paris”. Step 2: it proposes a period. New input: “The capital of France is Paris.”, and so on. Every generated token becomes context. (Given that input, Phi-3 does in fact pick Paris (Alammar & Grootendorst, 2024, pp. 80–81).)
Practical consequences. The sequence grows with every token, so the context window fills up (Alammar & Grootendorst, 2024, p. 22). To avoid recomputing everything at each step, intermediate results are stored in the KV cache (Alammar & Grootendorst, 2024, p. 83). The way each token is chosen, greedily or by sampling, changes the resulting text. And what has been written is not revised: a token once emitted conditions everything that follows.
One temptation is to conclude that an autoregressive model “does not plan” because it writes from left to right. The output format does not settle that question: what information about the continuation is represented internally at each step is an empirical question for interpretability. The GPT-2 and GPT-3 species are autoregressive; so was the recurrent decoder of Seq2seq (Alammar & Grootendorst, 2024, p. 12).