A transformer processes each token of the input along its own computational “lane” and has a maximum number of lanes: its context length. A model with a 4K context processes at most 4,000 tokens (Alammar & Grootendorst, 2024, p. 81). The limit covers both the prompt and the response: since generation is autoregressive generation, every token produced takes up room (Alammar & Grootendorst, 2024, p. 22). BERT worked with 512 tokens (Devlin et al., 2019, p. 4183); GPT-2 doubled its predecessor's context, from 512 to 1024 (Radford et al., 2019, sec. 2.3); recent models reach hundreds of thousands.
Tiny example. Assume, for simplicity, one word per token and a window of 8. In “Ann lent the book to Peter . He returned it yesterday”, on reaching yesterday the model only sees “book to Peter . He returned it yesterday”: for it, “Ann lent the” does not exist, and it could not say who lent the book.
Why is there a limit? Because self-attention compares each token with every other, and its cost grows with the square of the length, O(n² · d) per layer (Vaswani et al., 2017, sec. 4); hence variants such as sparse attention, which restricts which previous tokens are looked at (Alammar & Grootendorst, 2024, p. 96). Beyond the window there is no memory: the model keeps no state between calls, and assistants that “remember” a conversation send it the history again at every turn, in full or summarized (Alammar & Grootendorst, 2024, pp. 209–210). For long texts, semantic search and RAG systems split documents into chunks (Alammar & Grootendorst, 2024, p. 235).
The idea has an elegant precursor. In 1949, Warren Weaver imagined reading a book through an opaque mask with a slit: if the slit shows a single word, there is no telling whether fast means “rapid” or “motionless”; widen it to show N words on either side and, for large enough N, the ambiguity disappears (Weaver, 1949). The context window is that slit, now hundreds of thousands of tokens wide.