A model's vocabulary is the table of all the tokens its tokenizer can produce, each
with an integer identifier. Its size is a design decision: 30,522 entries in uncased BERT,
50,257 in GPT-2, 32,000 in Llama 2, a little over 100,000 in GPT-4
(Alammar & Grootendorst, 2024, pp. 47–55). Besides pieces of text it includes special tokens with no lexical
content: beginning and end of text, padding, unknown, mask ([MASK]) and, in chat models,
turn markers such as <|user|> and <|assistant|> (Alammar & Grootendorst, 2024, pp. 47–48, 54–56).
The vocabulary ties a model to its tokenizer: the model stores one embedding per entry, which is why a pretrained model cannot switch tokenizers without being retrained (Alammar & Grootendorst, 2024, p. 57). In Phi-3 that table is a 32,064 × 3,072 matrix, that is, about 98.5 million parameters devoted solely to representing tokens (Alammar & Grootendorst, 2024, p. 79). On the other side, the model's output is a distribution over exactly that set: it cannot predict anything outside it.
Tiny example. A whole-word vocabulary that contains apology does not know what to do with
apologetic if it never saw it in training: it must return [UNK]. A subword vocabulary with
apolog and suffixes such as -y, -ize, -etic and -ist covers those forms and any new
derivative (Alammar & Grootendorst, 2024, p. 44).
The linguistic temptation is to identify the vocabulary with a lexicon, but it is not one. It contains meaningless fragments, runs of whitespace (GPT-4 has a token for every run of up to 83 spaces) and programming keywords such as elif (Alammar & Grootendorst, 2024, p. 51). And it is closed: a language's lexicon is productive and open, whereas a model's vocabulary is fixed before training. Its openness to new words comes from combining existing pieces, not from adding entries.