Retrieval-augmented generation (RAG) is a system that searches before it writes. A retriever finds relevant passages in a document collection and a generator writes the answer by reading them. The term comes from P. Lewis et al. (2020), who presented it as a combination of two memories: a parametric one, stored in a model's weights, and a non-parametric one, an index of texts to be consulted (2020, sec. 1).
How it works
The retriever is a bi-encoder: one BERT-base encoder turns the question into a vector and another turns each document into a vector in the same space, as in Sentence-BERT. The December 2018 Wikipedia, cut into 21 million chunks of 100 words, is stored as an index, and retrieval amounts to finding the k vectors with the highest dot product (2020, secs. 2.2, 3). The generator is BART-large, a 400-million-parameter sequence-to-sequence task model that receives the question concatenated with each passage (2020, sec. 2.3).
The technical novelty is that the passage is treated as a latent variable: the system generates an answer from each of the k passages and combines them, weighted by relevance. In the RAG-Sequence variant the same passage supports the whole answer; in RAG-Token, each word can lean on a different one (2020, sec. 2.1). During fine-tuning the question encoder and the generator are trained together, without telling the system which document was the right one; the index stays fixed (2020, sec. 2.4).
What it achieved
RAG set the state of the art on three open-domain question-answering datasets (2020, sec. 1). With 626 million trainable parameters it scored 44.5 exact match on Natural Questions, against 28.9 for a comparably sized T5 that answers only from what is in its weights (2020, app. G). On a task of writing Jeopardy!-style questions, human raters judged RAG more factual in 42.7% of cases and BART in only 7.1% (2020, sec. 4.3). The most elegant experiment swaps the index: with the 2016 Wikipedia, RAG got 70% of 2016 world leaders right; with the 2018 one, 68% of 2018 leaders; with the mismatched index, only 4 to 12% (2020, sec. 4.5). Updating what it "knows" was as simple as changing its books.
From paper to design pattern
Today "RAG" mostly names a recipe: split documents into chunks, turn them into embeddings, store them in a vector database, retrieve the nearest neighbors of the query and paste them into the prompt of a generative model, which produces a "grounded" answer, ideally with citations (Alammar & Grootendorst, 2024, pp. 229–230, 249–251). Unlike the original, these systems do not usually train their parts jointly. Advanced versions rewrite the query, chain searches or let the model decide where to look, which brings them close to an agent (2024, pp. 255–256).
Limits
Dense semantic search always returns something, even when the collection lacks the answer; it fails on exact phrases, which call for keyword search; and it degrades outside the domain it was trained on (Alammar & Grootendorst, 2024, pp. 234–235). How to chunk documents is a design decision with no single recipe (2024, pp. 235–237). On some tasks the retriever "collapses," fetching the same documents every time, and the generator learns to ignore them (P. Lewis et al., 2020, app. H). RAG reduces hallucinations but does not eliminate them, and checking whether citations actually support what is claimed remains an open problem (Alammar & Grootendorst, 2024, p. 257).