ElemedexA pocket encyclopedia of AI architectures
ENES
ELMo
No. 026

ELMo

Embeddings from Language Models

Context

RecurrentEmbedding

It has no vector for each word, only one for each time a word occurs: the "bank" of a river and the "bank" that charges fees get different vectors. It reads every sentence twice, forwards and backwards. It was the first of the litter of models named after Sesame Street characters.

Height
2 biLSTM layers + character CNN
Weight
≈ 375 MB (fp32)
Parameters
93.6 M
Context
one sentence
Training data
1B Word Benchmark (≈ 30 M sentences, ≈ 800 M tokens)
Year
2018
Generation
4 · Transformers
Creators
Peters, Neumann, Iyyer, Gardner, Clark, Lee & Zettlemoyer (Allen Institute for AI / University of Washington)
Kind
Model

Base stats

Parameters
45
Data
52
Compute
43
Context
15
Generality
35
Transparency
35
Total
225

Approximate, ordinal scales meant for comparing species with each other; the hard numbers are in the data sheet.

Weaknesses

  • Shallow bidirectionality: two one-directional models concatenated, so no layer sees both sides at once.
  • It is recurrent: it reads one word at a time and makes poor use of GPU parallelism.
  • It is a feature supplier: every task still needs its own architecture on top.
  • It was trained on isolated, shuffled sentences, so it sees no context beyond the sentence.

Habitat

It grew up reading the 1B Word Benchmark, some 30 million sentences of English news shuffled one by one. Today you meet it mostly in the history of the field and in probing studies: pretrained transformers displaced it within months.

Moves

Abilities

One vector per occurrence
Each occurrence of a word gets a vector that depends on the whole sentence, not a fixed vector for the type.
Layer mixing
Each task learns how much to weight the character layer and each LSTM layer; lower layers contribute more syntax, upper ones more word sense.
Character input
A convolutional network over characters builds the initial representation, so there are no out-of-vocabulary words.

Anatomy

biLSTM layer×2“sat on the river bank”Character CNNForward LSTMBackwardLSTMWeighted sum of layers“bank” = riversideTask-specific model

ELMo, bottom to top. A character CNN produces an initial vector per word; two LSTM layers recontextualize it, one reading forwards and one backwards. In pretraining, the forward direction predicts the next word and the backward one the previous word. The ELMo vector is a weighted sum of the three layers, with weights learned by each task, and it is concatenated to the input of an existing model.

Evolution chain

  • context: one vector per occurrence, from a bidirectional LSTM019 word2vec
026 ELMo
  • bidirectional context, now with attention028 BERT

ELMo (Embeddings from Language Models) is a model that gives each word a different vector in each sentence where it occurs. It was introduced in 2018 by researchers at the Allen Institute for AI and the University of Washington (Peters et al., 2018), and it was the hinge between the static embeddings of word2vec and pretrained transformers.

The problem: one vector per word

word2vec gives "bank" a single vector, whether it is the bank of a river or the one that charges fees. Alammar & Grootendorst (2024) use exactly this example: the word's meaning, and therefore its embedding, should change with the context (2024, p. 11). ELMo solves the problem by moving from types to occurrences: it produces contextual embeddings (Alammar & Grootendorst, 2024, pp. 58–61).

How it works

Its core is a bidirectional language model (biLM). One LSTM reads the sentence left to right and predicts each word from the preceding ones; another reads it right to left and predicts each word from the following ones (Peters et al., 2018, pp. 2228–2229). Each direction has two layers of 4,096 units with 512-dimensional projections, and the input is not words but characters, processed by a convolutional network, so there are no unknown words (2018, p. 2230). It was trained for ten epochs on the 1B Word Benchmark, some 30 million sentences (2018, pp. 2228, 2230).

A word's ELMo vector combines all the layers (the character layer and both LSTMs) with weights learned by each task (2018, p. 2229). The biLM is then frozen and that vector is concatenated to the input of an existing task model: one for question answering, for inference, or for tagging (2018, pp. 2229–2230). In the terminology that BERT later popularized, it is a "feature-based" approach (Devlin et al., 2019, p. 4171).

What changed

Adding ELMo improved the state of the art on all six tasks evaluated (question answering, textual entailment, semantic roles, coreference, named entities and sentiment), with relative error reductions of 6 to 20% according to the authors (Peters et al., 2018, pp. 2230–2231). It also made models more data-efficient: for semantic roles, with 1% of the training data it matched the baseline trained on 10% (2018, p. 2234).

Most interesting for linguists is the layer-by-layer analysis. The nearest neighbors of play in GloVe mix forms and senses, mostly sporting ones; with ELMo, the play of a baseball report and the play on Broadway come apart (Peters et al., 2018, p. 2233). The first LSTM layer is better for part-of-speech tagging; the second, for word sense disambiguation (2018, pp. 2233–2234). Syntax below, semantics above, as in the linguistics textbooks.

Limits

ELMo's bidirectionality is shallow: two unidirectional models trained separately and concatenated, so no layer sees both sides at once. That was the criticism with which BERT introduced itself months later (Devlin et al., 2019, pp. 4172, 4178). Moreover, being recurrent, it reads one word at a time and cannot exploit GPU parallelism, and its corpus was shuffled at the sentence level, with no long-range structure (Radford et al., 2018, sec. 4.1). Its most visible legacy is another one: it started the habit of naming models after Sesame Street characters.

Sources

  • Alammar, J., & Grootendorst, M. (2024). Hands-on large language models: Language understanding and generation. O’Reilly Media. §
  • Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In J. Burstein, C. Doran, & T. Solorio (Eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) (pp. 4171–4186). Association for Computational Linguistics. https://doi.org/10.18653/v1/N19-1423 §
  • Firth, J. R. (1957). A synopsis of linguistic theory, 1930–1955. In Studies in linguistic analysis (pp. 1–32). Blackwell. §
  • Fodor, J. A., & Pylyshyn, Z. W. (1988). Connectionism and cognitive architecture: A critical analysis. Cognition, 28(1–2), 3–71. https://doi.org/10.1016/0010-0277(88)90031-5 §
  • Frege, G. (1884). Die Grundlagen der Arithmetik: Eine logisch mathematische Untersuchung über den Begriff der Zahl [The foundations of arithmetic: A logico-mathematical enquiry into the concept of number]. W. Koebner. §
  • Peters, M. E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018). Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) (pp. 2227–2237). Association for Computational Linguistics. https://doi.org/10.18653/v1/N18-1202 §
  • Radford, A., Narasimhan, K., Salimans, T., & Sutskever, I. (2018). Improving language understanding by generative pre-training [Technical report]. OpenAI. https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf §
  • Russin, J., McGrath, S. W., & Williams, D. J. (2025). From Frege to chatGPT: Compositionality in language, cognition, and deep neural networks (arXiv:2405.15164). arXiv. https://doi.org/10.48550/arXiv.2405.15164 §