An embedding assigns each item a point in a many-dimensional space, in the hope that the geometry reflects something of meaning: words used alike end up close, unrelated words far apart. Similarity thus becomes measurable with operations such as cosine similarity (Alammar & Grootendorst, 2024, pp. 8–10). The term comes from mathematics, where an embedding is a structure-preserving map.
Tiny example. In GloVe vectors trained on Wikipedia, the nearest neighbors of king are prince (similarity 0.82), queen (0.78), ii, emperor and son (Alammar & Grootendorst, 2024, p. 63). word2vec also popularized analogy arithmetic: vector(King) − vector(Man) + vector(Woman) lands closer to Queen than to any other word (Mikolov, Chen, et al., 2013, sec. 1.1).
Embeddings are not designed; they are learned. word2vec starts from random vectors and adjusts them while training a network to tell pairs of words that occur together from randomly drawn pairs; in the end, words with similar neighbors have similar vectors (Alammar & Grootendorst, 2024, pp. 64–67). It is the distributional hypothesis turned into a procedure. In an LLM, the first layer is a table with one embedding per token in the vocabulary, which also starts out random and is trained along with the rest of the model (Alammar & Grootendorst, 2024, pp. 57–58). There are also sentence and document embeddings, the basis of semantic search (Alammar & Grootendorst, 2024, pp. 61–62).
Two caveats. Dimensions rarely correspond to interpretable properties: picturing a “humanness” axis is a useful but false simplification (Alammar & Grootendorst, 2024, p. 9). And a static embedding gives a single vector per word, so the bank of a river and the bank that holds your money share one representation; contextual embeddings fix that. The embeddings demo lets you explore a toy space.