Before a model sees a text, a tokenizer splits it into tokens and replaces them with numeric identifiers; on the way out it does the reverse (Alammar & Grootendorst, 2024, pp. 38–43). There are four schemes: word tokens (used by word2vec; stumped by new words), character tokens (never fail, but lengthen sequences and force the model to learn spelling), byte tokens, and subword tokens, now dominant, with algorithms such as BPE (the GPT family) or WordPiece (BERT) (Alammar & Grootendorst, 2024, pp. 43–46; Sennrich et al., 2016).
A tokenizer depends on its method, on parameters such as the size of the vocabulary, and
on the data it was trained on, which is why tokenizers differ so much. The original uncased
BERT lowercases everything, loses line breaks, and turns emojis and Chinese characters into
[UNK], “unknown”; tokenizers built for code keep indentation whitespace
(Alammar & Grootendorst, 2024, pp. 48–56). The tokenizer demo lets you compare them.
Why LLMs struggle to count letters. GPT-4's tokenizer turns CAPITALIZATION into two tokens, CAPITAL and IZATION (Alammar & Grootendorst, 2024, p. 51). If we ask how many A's the word has, the model receives not letters but two numbers. It can only have learned the spelling of each token indirectly, for instance from texts in which words are spelled out; Edman et al. (2024) found that models seem to know how their tokens are spelled but make poor use of that knowledge when they have to manipulate text. Numbers fare similarly: in GPT-2, 870 is a single token but 871 is two (8 and 71), which is why some code models tokenize each digit separately (Alammar & Grootendorst, 2024, p. 52).
Tokenization is not neutral across languages either. The same text in translation can require up to 15 times more tokens depending on the language, which raises costs and shrinks the effective context window for some language communities (Petrov et al., 2023): an inequality settled before the model processes a single word.