A semantic search engine does not search documents but embeddings, and an embedding represents a short passage well and a book badly. There are two reasons: the models that produce them have a limited context window, and a single vector for a long document is a summary so compressed that it loses most specific information (Alammar & Grootendorst, 2024, pp. 235–236). Hence chunking: the index stores one vector per chunk, not per document, and semantic search returns the chunk closest to the question.
Strategies range from one sentence per chunk (too little context) to a paragraph or groups of three to eight sentences; one can also prepend the document's title to each chunk, or make chunks overlap so that they repeat part of the neighboring text (2024, p. 237). The original RAG system split Wikipedia into disjoint hundred-word chunks: 21 million "documents" (P. Lewis et al., 2020, sec. 3).
It looks like plumbing, but it is a semantic decision. A chunk that begins "That rule was repealed in 2019" is useless if the rule was named in the previous paragraph: the cut broke an anaphora. Chunks that are too small lose context; chunks that are too large dilute the vector and fill the prompt with noise. In RAG systems, many wrong answers come from a badly cut passage rather than from bad generation.
Linguists will hear a double echo. The word chunk already existed in psychology, in another sense: the unit of grouping in short-term memory (Miller, 1956). And the problem recalls Frege's context principle, which forbade asking for the meaning of a word in isolation: what is the smallest unit of text that keeps its sense outside its surroundings? Chunking answers by fiat, with a word count.