Thanks to residual connections, a token's vector is not replaced at each layer: things are added to it. In a pre-normalized transformer, each block computes
x ← x + Attention(LN(x)); x ← x + FFN(LN(x))
so the final vector is the initial embedding plus all the contributions of all the sublayers. Mechanistic interpretability calls that vector, running along the stack, one per position, the "residual stream". Alammar & Grootendorst (2024) put it more simply: each token travels its own computation lane, equally wide at input and output, and the lanes only communicate in attention (2024, pp. 81–82).
Why this is a useful way to think. Elhage et al. (2021) propose that all the model's components (embeddings, attention heads, feedforward layers, the output layer) communicate by reading from and writing to different subspaces of the residual stream, and that each attention head works as an independent operation whose result is added into the stream. This lets the model be decomposed into paths that can be studied separately (see circuits), and explains why it makes sense to read intermediate representations with probing, or to intervene by adding a vector to them, as in steering.
Where it is. In any transformer with residual connections: from GPT-2 and GPT-3 to LLaMA. It is not an extra part one can point to in the diagram, but a way of describing the part that is already there.
Caution. The picture of a "shared blackboard" or a "bus" is appealing, and it evokes global-workspace theories in psychology. But the stream is just a sum of vectors: its directions come unlabeled, many features share the same dimensions (superposition), and nothing guarantees that what the analyst reads is what the network "uses". It is a descriptive tool, not an organ.