In masked modeling, one takes a sentence, covers some of its tokens, and asks the
model to guess them. Nobody grades the answers: the correct one is already in the original
text, which is why this counts as self-supervised learning. BERT hid 15% of the
tokens, usually replacing them with a special [MASK] symbol (Devlin et al., 2019, p. 4174).
The cat [MASK] on the mat. → sat (high probability), slept, lay…
Unlike next-token prediction, which only lets the model look to the left, masking lets every position use the whole context, left and right. That yields better representations for classification or search, but a model trained this way cannot write fluently: it did not learn to continue a text, only to repair gaps (Alammar & Grootendorst, 2024, p. 19).
The idea has an explicit human precedent: the cloze test, used in the psychology of reading since the 1950s to measure comprehension by restoring deleted words (Taylor, 1953). That the same task serves both to assess readers and to train machines invites the question of what it actually measures: comprehension, or command of the distributional regularities of words?