A language model stores what it "knows" in its parameters: parametric memory, hard to update and hard to cite. P. Lewis et al. (2020) proposed combining it with a non-parametric memory: a dense vector index of Wikipedia split into 21 million hundred-word passages, which a neural retriever consults before the generator writes (2020, secs. 1, 3). One immediate advantage: updating what the system knows only requires replacing the index, with no retraining (2020, sec. 4.5).
Today the basic scheme has two steps (Alammar & Grootendorst, 2024, pp. 226–230, 249–250):
- Search. The question is turned into an embedding and the nearest passages are looked up in a vector database (semantic search), sometimes with a reranking stage.
- Grounded generation. The passages are pasted into the prompt along with the question, and the model is asked to answer on their basis, ideally citing them.
It is the standard technique for "chatting with your documents" and the industry's main defense against hallucination: it reduces errors and improves factuality (2024, p. 250). But it does not eliminate them: the retriever can fetch the wrong passage, and the model can ignore or distort what is in front of it. See Retrieval-augmented generation (RAG).
The distinction between parametric and non-parametric memory has a well-known philosophical echo. A. Clark & Chalmers (1998) proposed the extended-mind thesis with the example of Otto, who keeps in a notebook what others keep in biological memory. A RAG system is Otto with his notebook. Where, then, is the system's knowledge: in the weights, in the index, or in the coupling between them? And if the notebook says something false, is the error the model's?