Self-attention compares every token with every other, but it does not know in which order they came: without extra information, it treats the input as a set. "Dog bites man" and "Man bites dog" contain exactly the same tokens. Since the Transformer has no recurrence and no convolution, its authors had to "inject" position into each vector (Vaswani et al., 2017, sec. 3.5).
The original solution was to add to each embedding a vector of sines and cosines of different frequencies:
PE(pos, 2i) = sin(pos / 10000^(2i/d)), PE(pos, 2i+1) = cos(pos / 10000^(2i/d))
Each dimension is a wave; together they work like an odometer with many wheels. The authors chose this form because a shift by k positions is a linear function, which should make it easy to attend to relative positions. They also tried learned positions, with nearly identical results (Vaswani et al., 2017, secs. 3.5 and 6.2).
Learned positions. GPT-1 and BERT add to each token one learned vector per position, which fixes a maximum length: 512 tokens in both cases (Radford et al., 2018, sec. 4.1; Devlin et al., 2019, pp. 4174, 4183). Vision Transformer (ViT) does the same for image patches.
RoPE. Today rotary position embeddings dominate: instead of being added to the input, they rotate the query and key vectors by an angle proportional to position, right before they are compared, so that the score depends on the distance between tokens (Alammar & Grootendorst, 2024, pp. 102–104). LLaMA and its successors use them (Touvron, Lavril, et al., 2023, sec. 2.2). This also matters because training packs several documents into one window, and the model should not be told that a document's first token is number 50 (Alammar & Grootendorst, 2024, p. 103).
A linguistic angle. Here word order is not part of the structure of the computation, as in a grammar or a recurrent network: it is one more feature, added to each word's vector, that the model must learn to use. Buckner (2024) notes that splitting each vector into query, key and value and adding positional encodings were the two innovations that allowed recurrence to be abandoned (2024, p. 272). Whether syntactic compositionality can be rebuilt from that is an open empirical question.