TB: Retrieval-Augmented Generation

Overview

RAG (Retrieval-Augmented Generation) is a hybrid architecture that couples a pre-trained generative model with a queryable external knowledge base, allowing knowledge to live in an editable index rather than model weights. Introduced in Lewis et al. (NeurIPS 2020), RAG demonstrated that a 626M-parameter model with retrieval substantially outperforms a 770M-parameter model without it on knowledge-intensive tasks — making explicit that parametric memory is a poor substrate for factual knowledge. The core insight is architectural: separate what the model knows how to reason (parameters) from what it knows about the world (the index).

Core Tension

Static parametric memory vs. dynamic non-parametric memory. A pure LLM encodes knowledge at training time and cannot update it without retraining; it is compact, fast, and opaque. A pure retrieval system is transparent and updateable but cannot synthesize or generalize. RAG resolves this with a learnable bridge (the query encoder) that translates task context into retrieval queries, then conditions generation on retrieved passages. The unresolved tension is retrieval quality: the system can only be as factual as what it retrieves, and retrieval failures are silent.

Key Insights Across Sources

Parametric knowledge is a poor substrate for facts

  • Lewis et al. (NeurIPS 2020): T5-Large (770M params) achieves 28.9 EM on Natural Questions; RAG (626M params) achieves 44.5 EM — retrieval adds more than extra parameters.
  • Lewis et al. (NeurIPS 2020): "index hot-swapping" — replacing the Wikipedia index from 2016 to 2018 shifts factual answers appropriately, without any retraining. Knowledge lives in the index, not the weights.

The retriever is a learnable component, not a lookup table

  • Dense retrieval (DPR) outperforms BM25 (keyword search) on most tasks. The query encoder is jointly trained with the generator, so the system learns what kind of evidence is useful for the task. Only the document encoder is frozen during training — a pragmatic choice that avoids re-embedding 21M passages per gradient step.
  • Generator Filter: the DPR retriever and BART generator form an adversarial-cooperative pair: the retriever must produce passages the generator can use; the generator must learn to condition faithfully on what the retriever provides.

Grounded generation reduces hallucination

  • Human evaluation of Jeopardy question generation: RAG judged more factual than BART in 42.7% of cases vs. 7.1%. RAG also generates more distinct n-grams (83.5% vs. 70.7% on MS-MARCO), suggesting variety follows from factual diversity in retrieved passages, not from forcing it with decoding heuristics.
  • The hybrid also uses parametric knowledge as a fallback: RAG generates correct answers 11.8% of the time even when the answer is absent from all retrieved documents.

Two marginalization strategies with different tradeoffs

  • RAG-Sequence: one document per generation; efficient beam search; better for tasks where a single coherent source is sufficient (most open-domain QA).
  • RAG-Token: per-token marginalization over K documents; allows synthesis across sources; better for tasks requiring multi-hop reasoning. In practice, performance differences are small and task-dependent.

Related Concepts

  • Generator Filter — the retriever/generator pair is structurally a generator-filter: retriever generates candidates, generator filters and synthesizes
  • The Map is Not the Territory — parametric model weights are a lossy, frozen map of knowledge; RAG keeps the territory (the index) updatable
  • It Pays to Get the Design Right — RAG's architectural choice to separate retrieval from generation enables updatability and interpretability that brute-force scaling cannot provide
  • Abstractions Can Be Useful — the dense embedding space is an abstraction over document content that enables semantic (not just lexical) retrieval

Cross-Topic Connections

  • Agentic AI Systems — RAG is the standard pattern for giving agents access to domain knowledge; the index is the agent's external memory
  • AI and Experimental Particle Physics — RAG over CMS documentation or physics papers would allow HEP agents to ground responses in actual experiment documentation, not just parametric knowledge
  • Attention and Deep Work — ThirdBrain is a manually curated knowledge base; adding semantic retrieval over it would make /thirdbrain-query a RAG system
  • AI-Native Software Development — RAG over codebase documentation and API references is a primary use case in AI coding assistants

Backlinks