A bag of words reduces a text to an inventory: which words occur and how many times. Technically it is a multiset, because, unlike a set, it allows repetitions (Witten et al., 2017, p. 103). The procedure is simple: tokenize the text, build a vocabulary of all the distinct words, and turn each document into a vector with one dimension per word, whose value is the count (Alammar & Grootendorst, 2024, pp. 6–7).
Tiny example. "The dog bit the man" and "the man bit the dog" receive exactly the same vector: {bit: 1, dog: 1, man: 1, the: 2}. Millière & Buckner (2024a) point out that averaging the embeddings of a sentence's words, a common shortcut, is at bottom a bag as well: a law book and a book law come out identical (2024a, sec. 2.1).
Despite this blindness, it goes a long way. It underlies multinomial Naive Bayes for document classification (Witten et al., 2017, pp. 103–104) and the TF-IDF weighting of information retrieval, which rewards words that are frequent in a document but rare in the collection (2017, p. 314). It survives in topic modeling: BERTopic clusters documents with embeddings but describes each cluster with a weighted bag (Alammar & Grootendorst, 2024, pp. 148–149). And the CBOW architecture of word2vec (continuous bag of words) owes its name to averaging the neighbors regardless of their order.
The phrase has an ironic history. It appears in Harris (1954) to say the opposite of what it now names: language "is not merely a bag of words" but a tool whose properties were fashioned in the course of its use (1954, p. 156). The bag is the limiting case of a representation without compositionality, and the fact that it works so well for classifying by topic suggests that many tasks depend more on what is talked about than on what is said about it. A technical detail with a moral: reorder the input of an attention layer and its output is merely reordered the same way (Buckner, 2024, p. 23); without positional encoding, a transformer encoder could not tell who bit whom either.