Retrieval-Augmented Generation (RAG)

What Is Retrieval-Augmented Generation (RAG)?

Retrieval-augmented generation (RAG) is an architecture that searches a knowledge source for documents relevant to a query at request time and inserts them into the model's prompt, so the answer is grounded in retrieved evidence rather than only in what the model memorized during training. It gives models access to private, current, and verifiable information. The name comes from Lewis et al.'s 2020 NeurIPS paper at Meta AI, which set the state of the art on three open-domain question-answering tasks and produced language the authors measured as more specific, diverse, and factual than a parametric-only baseline [1].

Key Takeaways

• RAG separates knowledge from the model. The corpus can be updated, audited, and permission-scoped without touching the model, which training-time knowledge never allows.
• Answer quality is capped by retrieval quality. If the right passage never reaches the prompt, no amount of model capability recovers it.
• Grounding answers in retrieved text reduces hallucination and enables citations, but does not eliminate error: models can still misread or overrun their sources.
• The 2026-era pattern is increasingly agentic RAG, where the model drives search iteratively with tools instead of receiving one fixed batch of chunks.

How It Works

A RAG system has an offline half and an online half. Offline, documents are split into chunks, each chunk is converted into a vector embedding that encodes its meaning, and the vectors go into a vector database along with the text. Online, an incoming query is embedded the same way, the database returns the chunks whose vectors sit closest to the query, and those chunks are packed into the prompt with instructions to answer from the provided material. This is semantic search doing the fetching and the language model doing the reading and writing.

Production systems layer refinements onto that skeleton. Hybrid retrieval combines vector similarity with classic keyword scoring, since exact identifiers and rare terms defeat pure embeddings. A reranker re-scores the top candidates with a more expensive model before the final cut. Query rewriting expands terse questions, metadata filters enforce document permissions per user, and chunking strategy, sizes, overlaps, and respect for document structure quietly determines much of the ceiling. Answers typically cite which chunks they drew from, giving users a path to verify, which is the AI grounding property that makes RAG attractive for anything customer-facing. That appeal shows up in adoption data: Menlo Ventures' 2024 survey of 600 U.S. enterprise IT decision-makers found RAG in 51% of enterprise LLM architectures, up from 31% the year before [2].

The moving frontier is who controls retrieval. Classic RAG retrieves once, before generation, with fixed parameters. Agentic RAG hands the model a search tool and lets it query repeatedly, reformulate, follow references, and decide when it has enough evidence. Long context windows have not made either obsolete: corpora measured in gigabytes still cannot be inlined, and retrieval remains the cost- and relevance-filter that decides what deserves those context tokens.

Example

A B2B software company builds a support assistant over 4,000 help-center articles, release notes, and internal runbooks. The pipeline chunks documents by section, embeds them, and stores them with product-version metadata. When a customer asks "why did SSO stop working after the March update," the system embeds the question, filters to the customer's product version, retrieves the top eight chunks, reranks to three, and prompts the model to answer only from those passages with linked citations. The model surfaces a breaking change from the release notes plus the migration steps from a runbook. When the team ships fixes, they re-index the changed articles that night, and the assistant is current the next morning, with no model changes involved.

What People Get Wrong

The persistent mistake is treating RAG as insurance against hallucination. Retrieval narrows the space for invention but the model can still blend retrieved facts incorrectly, answer beyond what the sources support, or lean on parametric memory when retrieval comes back thin. Worse, weak retrieval produces confidently cited nonsense, since the citations point at chunks that never contained the claim. Trustworthy RAG requires evals on both halves: retrieval metrics for whether the right evidence arrived, and groundedness checks for whether the answer actually stayed inside it.

FAQ

What does RAG mean in the context of an LLM? A "rag llm" setup is simply a language model wired to a retrieval step: search runs first, the results go into the prompt, and the model answers from them. The model itself is unchanged; RAG describes the surrounding architecture, not a special kind of model.

Is RAG better than fine-tuning? They solve different problems. Retrieval augmented generation is the right tool for injecting facts, especially facts that change, differ per customer, or need citations. Fine-tuning is the right tool for teaching behavior, style, or task formats. Mature systems often use both: a fine-tuned model reading retrieved context.

Do long context windows make RAG obsolete? No. Even million-token windows cannot hold a serious corpus, attention degrades over very long inputs, and paying for a full corpus on every request is economically absurd. Long context changed the retrieval budget, letting systems pass larger and fewer chunks, rather than removing the need to retrieve.

Sources

1. Lewis et al., Meta AI. "Original RAG paper results on open-domain question answering." https://arxiv.org/abs/2005.11401. Accessed August 2026.
2. Menlo Ventures. "RAG adoption in enterprise LLM architectures, 2024 vs 2023." https://menlovc.com/2024-the-state-of-generative-ai-in-the-enterprise/. Accessed August 2026.