AI Architecture

Retrieval-Augmented Generation

Reference entry · last updated August 24, 2026

Retrieval-Augmented Generation (RAG) is an AI architecture that grounds a large language model's output in documents retrieved at query time, instead of relying only on knowledge encoded during training.[1] The model itself is unchanged; RAG is a system built around it. It addresses two related limits of a language model on its own — a fixed training cutoff, and no reliable way to signal "I don't know" — by handing the model real, current documents to answer from rather than asking it to answer from memory alone.[2]

The pipeline

RAG runs in four stages, in this order.[1][2]

Ingestion — source documents are split into chunks, each chunk converted to a vector embedding, and the chunk plus its vector stored in a searchable index. Runs offline, before any query arrives.
Retrieval — the user's query is converted to the same kind of vector, and the index is searched for the closest-matching chunks.
Augmentation — the retrieved chunks are inserted into the prompt alongside the original query.
Generation — the model produces an answer from that augmented prompt.

Retrieval itself runs one of three ways: dense (vector similarity, finds meaning but can miss exact keywords), sparse (keyword matching such as BM25, fast but misses semantic similarity), or hybrid (both, merged).[2]

Failure modes

Buried answers

Vector search finds chunks that read as similar to a query, not necessarily the chunk that answers it. Because documents are often organized hierarchically (title, then section, then sub-section), a query can match a document's title-level summary while the actual answer sits several sections deeper, worded differently enough that its similarity score to the query is too low to surface. The system does not retrieve the wrong document — it retrieves the right document at the wrong depth.

This specific failure pattern is widely discussed in RAG-engineering literature; no single canonical academic source names it, so it is described here rather than cited to one paper.

Index staleness

Building the index once is not sufficient. Source data changes after ingestion runs, and a stale index returns outdated answers with the same apparent confidence as fresh ones. Keeping the external data current, through automated real-time syncing or scheduled batch refreshes, is a distinct, ongoing pipeline requirement separate from the one-time ingestion build.[1]

Variants

GraphRAG

A second retrieval mode layered on top of vector search: a knowledge graph encoding explicit relationships between documents or entities, which the system can traverse instead of only measuring similarity. Vector search asks what looks similar to a query; graph traversal asks what is structurally connected to an entity — a different operation, useful for relationship-dependent or multi-hop queries that similarity search handles poorly.

Reranking

A second relevance pass over the initial retrieval shortlist, not the whole index. Initial retrieval is cheap and runs over the full corpus to return an approximate top-K; a separate, more expensive model (often a cross-encoder, which scores the query and each candidate document together rather than comparing precomputed vectors) re-scores that shortlist and reorders it before anything reaches the prompt. The two steps trade speed for precision on purpose.

References

^ "What is RAG (Retrieval-Augmented Generation)?" — Amazon Web Services — https://aws.amazon.com/what-is/retrieval-augmented-generation/
^ Jenna Pederson, "Retrieval-Augmented Generation (RAG)" — Pinecone, 2025-06-12 — https://www.pinecone.io/learn/retrieval-augmented-generation/