Donahue et al. (2020) define infilling as predicting missing spans anywhere in a document, so that they are consistent with both the preceding and the following text. Ordinary language modeling is the special case in which the gap is at the end (2020, p. 2492). Their example: "She ate [blank] for [blank]" → "She ate leftover pasta for lunch."
The problem includes deciding how much text goes into each gap. BERT fills positions
covered by [MASK], but its bidirectional attention ties it to gaps of known length
(2020, p. 2492); T5 replaces whole spans with sentinel
tokens and its decoder writes only what is missing
(Raffel et al., 2020, pp. 12–13). An autoregressive generation model, by contrast, conditions only on what
lies to its left.
The practical fix was to reorder the data, not the architecture. Donahue et al. (2020) train GPT-2 on the perforated text, a separator and then the answers (2020, sec. 3). Bavarian et al. (2022) cut each document into prefix, middle and suffix, and present it as prefix, suffix and middle, with special markers; if half of the data is transformed this way, the model learns to infill without losing its ability to write left to right ("FIM for free") (-@bavarian2022, secs. 1, 3). The motivation is code: a programming assistant that writes a function's documentation or completes one already begun, with code before and after the cursor (2022, sec. 1). Diffusion language models need no detour: having no causal mask, they infill by conditioning on any set of positions (Lou et al., 2024, sec. 4.2).
For linguists, the precedent is the cloze test (Taylor, 1953). Donahue et al. (2020) present infilling as its generalization, since cloze historically meant restoring single words (2020, p. 2492, n. 1). What is instructive is the asymmetry: for a reader, the right context helps; for a left-to-right generator, that help is invisible until it is taught, with the same data, to read them in another order.