A special token is a vocabulary entry that corresponds to no piece of text but to a
function: marking where a sequence begins or ends, separating two texts, padding empty
positions, hiding a word, or indicating who is speaking (Alammar & Grootendorst, 2024, p. 47). The
tokenizer of BERT has five: [CLS], [SEP], [PAD], [UNK] (for
whatever it cannot encode) and [MASK] (2024, pp. 47–48); that of
GPT-2 has only one: <|endoftext|> (2024, p. 49).
Tiny example. To decide whether one sentence entails another, BERT receives
[CLS] the dog is asleep [SEP] the animal is resting [SEP]
[SEP] separates the sentences (together with an embedding marking which one each token
belongs to), and the final vector of [CLS] summarizes the whole pair for classification
(Devlin et al., 2019, pp. 4174–4175). [MASK] raises a curious problem: it appears in pretraining
but never in fine-tuning, which is why BERT uses it in only 80% of the positions chosen
for hiding (2019, p. 4174) (see masked language modeling).
The assistants of 2023 brought turn tokens: <|user|>, <|assistant|>, <|system|>
(Alammar & Grootendorst, 2024, p. 54). Phi-3's chat template wraps each exchange as <s><|user|> …
<|end|><|assistant|>. It is the format the model was trained on, and it tells the model who
said what and when to stop: for the model, ending a turn means predicting <|end|>
(2024, p. 169).
These tokens have no lexical content; their vectors mean whatever training makes them mean.
Without fine-tuning, the [CLS] vector is a poor summary of the sentence, worse than
averaging GloVe vectors (Reimers & Gurevych, 2019, p. 3982): an extreme case of meaning as functional
role, not unlike the auxiliary symbols of a formal language. And the template reveals
something uncomfortable: in a chat, "who is speaking" is not an external fact but one more
token in the same sequence as everything else. The boundary between the operator's
instructions and the user's text is a learned convention, not a barrier (see
system prompt and jailbreak).