The causal mask operates inside self-attention: before the softmax, the scores of positions after the current one are replaced with −∞, so their weight comes out exactly 0 (Vaswani et al., 2017, sec. 3.2.3). Seen as a matrix, only the lower triangle remains visible. For the sentence "The cat sat down":
The cat sat down
The ✓ · · ·
cat ✓ ✓ · ·
sat ✓ ✓ ✓ ·
down ✓ ✓ ✓ ✓
Why it exists. A generative language model is trained on next-token prediction. Thanks to the mask, a single pass over a text of n tokens trains n predictions at once, none of them "peeking" at the answer: without the mask, predicting sat while looking at sat would be trivial (Alammar & Grootendorst, 2024, p. 17; Buckner, 2024, p. 274). At generation time the issue does not arise (the future does not exist yet), but the mask ensures the model runs exactly as it was trained.
Who has it. The decoder of the original Transformer; and the whole family of decoder-only models, from GPT-1 ("masked self-attention heads", (Radford et al., 2018, sec. 4.1)) to GPT-3, LLaMA and ChatGPT (InstructGPT). BERT, by contrast, does not use it: it can look both ways, hence the B for bidirectional (Alammar & Grootendorst, 2024, p. 97). That is the technical difference between models that write and models that only represent.
A misleading name. "Causal" has nothing to do with causes: it only means that temporal order is respected. "Autoregressive mask" would be more accurate.
A linguistic angle. Humans also process language incrementally, and there has been debate about whether unidirectional attention is more plausible than bidirectional. Buckner (2024) reviews studies in which GPT-family models best predicted activity in left-temporal language areas, albeit with metrics whose epistemology is still disputed (2024, pp. 276–277).