Retrieval-augmented generation

Retrieval-augmented generation, or RAG, is a design pattern in which a language model answers a query with the help of documents fetched at the moment of the request. A retrieval component searches an index and returns a small set of passages; a generation component receives the query and those passages and produces an answer conditioned on them 1.

The pattern addresses a basic property of language models: their knowledge is frozen at the end of training and is stored implicitly in their weights. Retrieval adds a second, explicit memory that can be updated by editing a database rather than retraining a model, and it can expose the sources behind an answer.

This entry covers the original architecture, the pipeline that most systems now use, the reasons retrieval became standard in enterprise applications, the failures that persist, and the argument over how much it actually solves.

A language model's knowledge is parametric: it is distributed across billions of weights and cannot be inspected or edited directly. Updating it requires further training, which is expensive, risks degrading other capabilities, and cannot keep pace with information that changes daily.

A document index is non-parametric memory. It stores text and metadata outside the model, can be rebuilt or patched, and supports exact citation. A large index can hold millions of passages and be refreshed hourly, which is not possible for weights. The trade-off is that the model must be given the relevant text at inference time, and the quality of the answer then depends on the quality of the retrieval.

Earlier systems chose between the two. A model could be fine-tuned on a corpus, or a search engine could return passages for a reader to interpret. RAG joined them: the retriever supplies knowledge, the generator supplies language and reasoning, and both are trained or prompted to work together 1 2. In practice the boundary is now blurred, since many systems combine a fine-tuned generator with a live index and treat each as a separate component.

The 2020 RAG paper combined a dense retriever with a sequence-to-sequence generator. The retriever encoded the query and each passage as vectors and selected the passages with the highest similarity; the generator was a pre-trained encoder-decoder that conditioned on the query and the retrieved text 1. The index in the original experiments held roughly 21 million Wikipedia passages, searched with a maximum inner product lookup.

The retriever was a dual encoder of the kind introduced for dense passage retrieval, trained so that questions and passages containing their answers were close in vector space. Dense retrieval could match paraphrases that keyword search missed, and it outperformed sparse retrieval on open-domain question answering 3.

The paper proposed two ways to use the retrieved passages. RAG-Sequence conditions on the same set of documents for the whole answer, while RAG-Token can attend to different documents at each generated token. The model was trained end to end, with gradients flowing through the retriever, so that retrieval was optimised for the generation task rather than for lexical similarity.

That end-to-end training is the part most production systems have abandoned. In practice the retriever is often a separate, frozen embedding model, and the generator is a general-purpose language model that is prompted rather than fine-tuned.

A contemporary RAG system has an offline and an online phase. Offline, documents are parsed, split into chunks, embedded with a model, and written to an index that supports approximate nearest-neighbour search. Metadata such as source, date, and access permissions is stored alongside the vectors.

Online, the query is embedded and used to retrieve a candidate set, often combining dense similarity with lexical scoring so that exact names and identifiers are not missed. A reranker, usually a cross-encoder that scores query-passage pairs directly, reorders the candidates. The top passages are assembled into a prompt with instructions and, where possible, source markers 4.

Generation is the last step, and it is where faithfulness is decided. The prompt must tell the model what to do when the passages do not contain an answer, and the output must be checked against the passages if citations are to mean anything. Many systems add a post-generation verification step that tests whether each claim is supported by the retrieved text.

Around this loop sit query rewriting, multi-query expansion, caching, and access control. Each is a place where the system can fail, and each adds latency and cost that must be weighed against the improvement in answer quality 5.

RAG made language models usable on private and current information without retraining. An organisation can point a system at its own manuals, tickets, contracts, or policies, rebuild the index when documents change, and avoid the cost and risk of fine-tuning a model on internal data.

It also made answers auditable. A response that cites the passages it used can be checked by a person, which matters in law, medicine, customer support, and any setting where an unsupported claim is costly. The citation is only as good as the retrieval, but the architecture at least creates a place for evidence to appear.

The economics favoured it. Serving a retrieval query and a shorter prompt is often cheaper than processing a very long context, and the index can be updated without touching the model. For a generation of enterprise products, RAG was the difference between a demonstration and a deployable system 4. Customer support, internal search, and coding assistants were rebuilt around it, and a market for vector databases, embedding models, and retrieval services grew alongside the models. It also moved the unit of deployment from a model to a pipeline, which changed what teams had to monitor.

It arrived at the right moment. Language models had become good enough to synthesise passages fluently, and vector search had become cheap and easy to operate. The combination produced a standard architecture before there was a standard name for the products built on it.

Retrieval can fail before generation begins. The relevant document may not be in the index, the chunking may have separated an answer from its context, the embedding may miss a paraphrase, or the reranker may discard the one passage that mattered. A 2024 case study grouped these into missing content, missed top-ranked documents, and passages that never reach the context window 5.

Even when the right passage is present, the model may not use it well. Information in the middle of a long context is used less reliably than information at the beginning or end, a pattern documented across models and tasks 6. The model may also prefer a fluent prior over an awkward but correct passage, producing an answer that contradicts its own sources.

Chunking is a persistent design problem. Small chunks retrieve precisely but lose context; large chunks preserve context but dilute the embedding and consume the window. Hierarchical retrieval, parent-document retrieval, and sentence-window methods are attempts to get both, and none is universally best.

Multi-hop questions expose the limits of single-shot retrieval. A question that requires combining two documents may retrieve one and miss the other, and a pipeline that retrieves once and generates once has no way to notice. Agentic retrieval, which lets the model issue several searches, addresses this at the cost of latency and more failure modes.

RAG systems are evaluated at two levels. Retrieval is measured with recall at k, mean reciprocal rank, and normalised discounted cumulative gain, which ask whether the relevant passage was found and how highly it was ranked. Generation is measured for faithfulness, answer relevance, and completeness.

Faithfulness is the hard one. A model-graded judge can check whether each claim in an answer is entailed by the retrieved context, and frameworks such as RAGAS automate the process without human annotations 7. The approach is cheap and useful, and it inherits the judge model's biases, including a tendency to accept fluent paraphrase as support.

End-to-end evaluation is harder still. A system can score well on retrieval and poorly on answers, or produce correct answers from the model's prior while the retrieved context is irrelevant. Separating the contributions requires ablations that most published evaluations do not report.

Benchmarks such as KILT and BEIR provide comparable retrieval and generation tasks, but they are static. Production corpora change, queries reflect a particular business, and the distribution of questions is rarely represented in a public benchmark. Human evaluation remains the reference standard, and it is expensive and inconsistent; most published comparisons rely on model judges instead. Internal evaluation sets remain the only reliable measure, and they are rarely published.

Retrieval expands the attack surface. Any document that enters the index is a potential carrier of instructions, and a language model does not reliably distinguish data from commands. An attacker who can place text in a retrieved document can attempt to redirect the model's behaviour, a class of attack known as indirect prompt injection 8.

Access control must be enforced at retrieval time, not at generation time. If a user's query can retrieve a document they are not entitled to see, the model may summarise it, quote it, or leak it through a citation. Permission-aware indexing, filtering before generation, and per-user namespaces are the standard mitigations.

Poisoning is a related risk. An index built from user-generated content, web pages, or shared drives can be manipulated so that malicious passages rank highly for common queries. Detection is difficult because the passage may be factually plausible and the manipulation may be subtle.

Finally, logging and retention create their own exposure. RAG systems often store queries, retrieved passages, and answers, and those records can contain personal or confidential information that outlives the request. Defence in depth — allowlists, sandboxed tools, and human approval for consequential actions — is the current best practice, and it is a mitigation rather than a fix.

The central disagreement is whether RAG reduces hallucination. Vendors and many practitioners describe it as the standard remedy: grounding answers in retrieved text makes fabrication less likely and makes errors traceable. Researchers point out that grounding changes the failure mode rather than eliminating it, since models can misread, overgeneralise, or ignore the passages they are given 5.

The second disagreement concerns long context. As models accept hundreds of thousands of tokens, some argue that retrieval will be absorbed into context: put the whole corpus in the prompt and let attention do the work. Others note that cost, latency, and the middle-of-context degradation make this impractical for large corpora, and that retrieval is a form of compression that remains necessary 6.

The third concerns architecture. The original RAG trained retrieval and generation together; most deployed systems do not. Whether end-to-end training produces meaningfully better answers, or whether a good embedding model and a strong generator are sufficient, is unresolved and depends on the task.

A fourth dispute is about attribution. A citation can be genuine, in the sense that the passage exists and was retrieved, without actually supporting the sentence it is attached to. Systems that display sources without verifying entailment can create an impression of rigour that the underlying answer does not have. The newest version of the argument is that agentic search, in which the model decides when and what to retrieve, will replace the fixed pipeline; both patterns are currently deployed, and the evidence for either is thin.