BERT is a transformer encoder trained to represent text, not to write it. It takes a sequence of up to 512 tokens and returns, for each of them, a vector that depends on the whole sentence. Alammar & Grootendorst (2024) present it as the first great "representation model," as opposed to the generative models of the GPT family (2024, pp. 18–20).
How it works
The architecture is almost exactly the left half of the original transformer: a stack of self-attention and feedforward (MLP) block blocks, with residual connections and normalization (Devlin et al., 2019, p. 4173). The base version has 12 layers, 768-dimensional vectors and 12 attention heads (110 million parameters); large has 24 layers and 340 million.
The novelty lies in how it is trained. A classic language model predicts the next word, so it can only look to the left. BERT is instead pretrained on two tasks (Devlin et al., 2019, pp. 4174–4175):
- Masked language modeling (MLM): 15% of the tokens are
chosen at random and hidden (usually replaced by a special
[MASK]token); the model must reconstruct them from the context on both sides. The authors explicitly connect this with the cloze test from the psychology of reading (Taylor, 1953): the school exercise of filling in the gaps of a text. - Next sentence prediction: given two sentences, decide whether the second one followed the first in the original text or was picked at random.
The result is a model that "knows" no task in particular but can be adapted to many. To
classify reviews, for instance, one takes the vector of the special [CLS] token, adds an
output layer, and runs a short fine-tuning on labeled examples
(Alammar & Grootendorst, 2024, pp. 18–19). This recipe (pretrain once, fine-tune many times) is the
transfer learning paradigm that dominated language processing from 2018 to 2020.
What changed
Compared with ELMo, which concatenated two LSTMs (one reading left-to-right, the other right-to-left), BERT is deeply bidirectional: at every layer, every word sees every other word. Compared with GPT-1, which uses the same kind of stack but with a causal mask, BERT gives up generation in exchange for better representations. In 2018 it set the state of the art on eleven understanding tasks, including the GLUE benchmark and the SQuAD question-answering dataset (Devlin et al., 2019, p. 4171).
Forms and offspring
Like some Pokémon, BERT has regional forms: RoBERTa (more data, no next-sentence prediction), DistilBERT (a smaller, distilled version), and multilingual variants. Its most widely used evolution today is Sentence-BERT, which produces one vector per sentence for semantic search.
The family grew again just when text generation seemed to have taken all the attention. ModernBERT brings recent models' improvements to the encoder and is trained on 2 trillion tokens with a native window of 8,192 (Warner et al., 2025); mmBERT extends the recipe to more than 1,800 languages (Marone et al., 2025). The Ettin suite trains pairs of encoders and decoders on the same data with the same recipe, from 17 million to 1 billion parameters: encoders win at classification and retrieval, decoders at generation, and retraining one on the other's objective does not close the gap. A 400 M encoder beats a 1 B decoder at inference (MNLI) (Weller et al., 2025).
Limits
The 512-token window is short, and the [MASK] token creates a mismatch: it appears in
training but never when the model is used (Devlin et al., 2019, p. 4174). More serious, from an
epistemic point of view, is that its benchmark success turned out to be partly illusory:
several studies showed that it exploited superficial regularities in the data (see the
professor's note).