A language model reads neither letters nor words: it reads tokens, pieces of text that a tokenizer cuts out and replaces with integers. The instruction “Write an email apologizing to Sarah…” reaches the Phi-3 model as a list of identifiers (1, 14350, 385, 4876, …), each of which points to an entry in the vocabulary (Alammar & Grootendorst, 2024, pp. 40–41). Tokens are the model's input and also its output: text is generated one token at a time (Alammar & Grootendorst, 2024, p. 38).
Tiny example. That tokenizer splits apologizing into apolog + izing, tragic into
trag + ic and mishap into m + ish + ap; it also adds special tokens with no
lexical content, such as <s> to mark the beginning of the text (Alammar & Grootendorst, 2024, pp. 41–42).
For a linguist it helps to keep three notions apart. A token is not a word: a word may take up several tokens, and a token may include the space before it. Nor is it a morpheme: mis-hap has a morphological boundary the tokenizer ignores, while m|ish|ap respects none. Tokens are substrings that were frequent in the data the tokenizer was trained on, chosen by statistical or compression criteria (see BPE); when they coincide with morphemes, it is because morphemes are frequent, not because the algorithm knows morphology. In English, a subword token averages about three characters (Alammar & Grootendorst, 2024, p. 45).
A terminological note for philosophers: machine-learning jargon calls both the vocabulary entry (what Peirce would call a type) and each of its occurrences in a text (a token in Peirce's sense) a “token”. The distinction matters: static embeddings are assigned to types, contextual ones to occurrences. The notion has also spilled beyond text: the Vision Transformer treats image patches as if they were tokens (Alammar & Grootendorst, 2024, p. 262).