A static embedding gives bank a single vector, whether it is the side of a river or a financial institution (Alammar & Grootendorst, 2024, p. 11). A contextual embedding, by contrast, is computed for each occurrence: the model reads the whole sentence and produces one vector per token that already incorporates information from its surroundings (Alammar & Grootendorst, 2024, pp. 58–61). It was introduced at scale by ELMo, with bidirectional recurrent networks (Peters et al., 2018), and transformers such as BERT made it the norm.
Tiny example. A small BERT-like model processes “Hello world” and returns a tensor of shape
[1, 4, 384]: four vectors of 384 numbers, one per token ([CLS], Hello, world, [SEP])
(Alammar & Grootendorst, 2024, p. 60). In “we sat on the river bank” and “the bank charged me a fee”, the
vector for bank comes out different each time, even though both start from the same row of
the embedding table.
The mechanism that mixes in context is self-attention: each word's representation is modified by weighting the others according to their relevance. Buckner (2024) illustrates it with a case of long-distance disambiguation: in a paragraph that begins with someone fishing and ends with “she stepped away from the bank”, the distant cues must push the vector for bank toward the region of semantic space that corresponds to rivers (2024, pp. 270–273).
For philosophers of language there is a natural reading: static embeddings represent types, contextual ones occurrences. But one should not simply identify different vectors with different senses. Contextual vectors vary continuously, also with topic, register and syntactic position, and they do not come segmented into senses like a dictionary. Whether that variation encodes polysemy, homonymy or something more diffuse is an empirical question, studied with probing.