← ELI5 · Nestor G Pestelos Jr

Retrieval-Augmented Generation

How does an AI answer things it wasn't taught?

The fancy term is RAG. The idea underneath is one you already use: look it up before you answer.

The big word, unpacked

RAG splits into Retrieval, Augmented, and Generation, feeding into one answer RETRIEVAL find the document AUGMENTED add it to the prompt GENERATION write the answer one grounded answer

Three ordinary steps. Nothing about "RAG" is mystical once it's spelled out.

The problem: its memory has a closing date

A timeline with facts before the training cutoff and blank question marks after it before training ended facts it learned TRAINING STOPS HERE after that date ? ? ?

It doesn't leave that second box empty on the page — it fills it in and sounds sure.

Why it doesn't just say "I don't know"

An AI writes one likely next word at a time. There is no built-in switch that says "stop, I have nothing here" — the most likely next word is still a confident-sounding one, even when nothing behind it is true.

The fix: look it up, then answer

A question is turned into a search, matching documents are found, handed to the model, and it answers from them your question search the document store matching pages AI reads those pages, then writes the answer from them

Same AI, same brain — it's just holding the right page open while it writes.

The library gets built ahead of time

A stack of documents is chopped into small pieces and filed on a searchable shelf, ahead of any question documents pieces SHELF, FILED BY MEANING

This part happens once, in advance — not while you're waiting for an answer.

It can still grab the wrong page

A question about API timeout matches the title-level summary document, while the real answer sits buried five sections deeper and is never picked Question: "how do I configure the API timeout?" retrieved: "API General Settings" (title match) but no timeout value in it never picked: Section 5, buried — "set timeout_ms to 5000"

It didn't find the wrong document — it found the right one, at the wrong depth.

And finding the right page still isn't a guarantee

Even when the lookup grabs the exact right page, the AI can still write an answer that contradicts it. Two different things have to go right: finding the correct source, and actually being faithful to it once it's found.

Sources: AWS — What Is RAG?, Pinecone — Retrieval-Augmented Generation. The API-timeout example is illustrative, not a documented incident.

Back to top