word2vec is a method for turning every word in a vocabulary into a vector of a few hundred numbers, so that words appearing in similar contexts end up close to one another. It neither writes nor classifies: its product is a table of embeddings, one row per word, meant as raw material for other systems. The name covers two architectures that Mikolov and colleagues presented at Google in 2013, CBOW and skip-gram (Mikolov, Chen, et al., 2013).
How it works
A window slides over the text. In CBOW (continuous bag of words), the vectors of the neighboring words are averaged and the model tries to guess the word in the middle; since they are averaged, order does not matter: hence the "bag of words." Skip-gram does the reverse: from the middle word it predicts the neighbors (Mikolov, Chen, et al., 2013, secs. 3.1–3.2). Nobody labels anything: the text itself supplies the answers (self-supervised learning).
The decisive move was to take things away. The neural language model of Bengio et al. (2003) learned word vectors together with a nonlinear hidden layer, where almost all the computation went. Mikolov and colleagues removed it: a less expressive model, but one that could be trained on vastly more data (2013, sec. 3). A few months later the same team also replaced the expensive softmax over the whole vocabulary with a yes-or-no question: is this word a real neighbor or one drawn at random? This negative sampling is an early form of contrastive learning (Alammar & Grootendorst, 2024, pp. 64–65, 293). In the end, the prediction task is thrown away and the input weight matrix is kept: each row is the vector of one word (M. Mitchell, 2019, ch. 11).
The corpus was Google News, about 6 billion tokens, with a vocabulary of one million words (Mikolov, Chen, et al., 2013, sec. 4.2); the best skip-gram (1,000-dimensional vectors) trained in about two and a half days on ≈ 125 CPU cores (2013, sec. 4.4). The team then released the code and vectors trained on more than 100 billion words (2013, sec. 7).
What changed
In a one-hot encoding, each word is an arbitrary index: nothing says that hated and abhorred are alike. With word2vec, likeness is an angle between vectors (cosine similarity), and the neighbors of France turn out to be other European countries although nobody ever taught it what a country is (M. Mitchell, 2019, ch. 11). A space of this kind can be explored in the embeddings lab.
What drew the most attention were the linear regularities: vector(biggest) − vector(big) + vector(small) lands near smallest, and something similar happens with countries and capitals (Mikolov, Chen, et al., 2013, sec. 4). To measure this, the authors built a benchmark of 8,869 semantic and 10,675 syntactic questions (2013, sec. 4.1). (The famous king − man + woman ≈ queen came from earlier work with vectors taken from a recurrent network (2013, secs. 1.1, 4.3).)
Until 2018, vectors like these were the usual input of nearly every language system (M. Mitchell, 2019, ch. 11); the encoders of Seq2seq could take them as input (Alammar & Grootendorst, 2024, p. 12), and the recipe was exported to other domains, such as recommending songs by treating them as words (2024, p. 67). Levy & Goldberg (2014) showed that skip-gram with negative sampling implicitly factorizes a matrix of co-occurrences weighted by mutual information: the old count-based distributional semantics in new clothes.
Limits
- One vector per word. "Bank" is the same by the river and in the financial system (Alammar & Grootendorst, 2024, p. 11); fixing that was the job of the contextual embeddings of ELMo.
- No order, no compositionality. The team's second paper admits that Air Canada cannot be obtained by combining Air and Canada (Mikolov, Sutskever, et al., 2013).
- The fine print of analogies. The search discards the words in the question, and only an exact match counts (Mikolov, Chen, et al., 2013, secs. 4–4.1). Without that exclusion, the answer is almost always a query word, often king itself; and part of the success can be had just by returning the nearest neighbor of the third word (Linzen, 2016; Nissim et al., 2020). There are errors like Japan: sushi → France: tapas (2013, sec. 5) or fish is to water as bird is to hydrant (M. Mitchell, 2019, ch. 11).
- Biases. Man is to computer programmer as woman is to homemaker (M. Mitchell, 2019, ch. 11). But since the exclusion rule forbids answering programmer, part of the effect is an artifact of the method (Nissim et al., 2020): the bias is real, but analogies are not the best instrument for measuring it.