If a relation (singular–plural, country–capital) were encoded as a constant offset in embedding space, adding it would take you from one pair to the other. To answer "big is to biggest as small is to ?", Mikolov, Chen, et al. (2013) compute vector(biggest) − vector(big) + vector(small) and return the closest word by cosine similarity, discarding the question words (2013, sec. 4). With this test (8,869 semantic and 10,675 syntactic analogies, counting only the exact word as a hit) they evaluated word2vec; the best model reached 65.6% (2013, secs. 4.1, 4.4).
There are precedents in psychology: Rumelhart and Abrahamson (1973) modeled analogies between animal names as parallelograms in a similarity space. And there is fine print. Levy and Goldberg (2014) showed that, with normalized vectors, the method amounts to adding and subtracting similarities (hence the name 3CosAdd): cos(x, B) − cos(x, A) + cos(x, C). Linzen (2016) went further: without excluding the question words, the answer was almost always C; and part of the success can be had by simply returning the nearest neighbor of C. Nissim et al. (2020) noted that the same rule forbids answering doctor to "man is to doctor as woman is to ?", which exaggerated some famous biases.
For M. Mitchell (2019), who studied analogy with Hofstadter, these games fall short (2019, ch. 11). Genuine analogy requires conceptual slippage: if abc changes to abd, what happens to xyz? Whoever answers wyz has let "last" slip to "first" and "successor" to "predecessor" (2019, ch. 14). A fixed vector offset cannot slip: it applies the same subtraction in every case. The question is whether the contextual representations of current models, which are no longer fixed, change that diagnosis.