How RAG Actually Works: AI That Reads Before It Answers
How RAG Actually Works: AI That Reads Before It Answers
Ask a chatbot about your company's refund policy and it will answer confidently — even though it has never seen the policy. RAG fixes that by making the model look things up first.
Picture a support chatbot at an online store. A customer asks, "Can I return these headphones after 30 days?" The bot replies, smoothly and confidently, that returns are accepted within 30 days.
There is just one problem: the store's real policy allows 14 days. The bot never saw the policy. It guessed, because guessing fluently is what language models do.
Retrieval-augmented generation — RAG for short — is the fix most real products use. Before answering, the system searches your own documents, finds the passages that matter, and hands them to the model. The model then answers from those passages, not from memory.
The open-book exam idea
The term comes from a 2020 paper by Patrick Lewis and colleagues at Facebook AI Research, with collaborators at University College London and New York University. Their system paired a language model with an external index of Wikipedia and answered questions from what it retrieved.
The paper's framing is still the clearest way to think about it. A language model has parametric memory — knowledge baked into its weights during training. RAG adds non-parametric memory — knowledge stored outside the model, in documents it can search at answer time.
That split is the whole point. Swap the documents and you change what the system knows, without retraining anything. As IBM's RAG pattern documentation puts it, when new information becomes available, all that's needed is to augment the model's external knowledge base.
First, you build the book
Everything before a question is asked is called indexing, and it happens once — then again whenever your documents change. It has three steps, and the quality of all three decides the quality of every answer.
Step 1: chunking. Documents get split into small passages, because a whole 500-page manual is useless as a search result. A practical starting point, per StackAI's RAG guide, is 400–700 tokens per chunk with 10–15% overlap, so a key sentence is never cut in half at a boundary.
Good chunks also carry their context with them: the document title and section heading at the top. Without that, a retrieved passage like "returns are accepted within 14 days" is an orphan — true, but impossible to interpret or cite properly.
Step 2: embeddings. An embedding model turns each chunk into a vector — a long list of numbers that captures what the text means. Two passages with similar meanings end up with similar numbers, even if they share no words. "Refund window" and "return deadline" land close together.
Step 3: storage. Those vectors go into a vector database (a dedicated one like Pinecone or Qdrant, or an extension like pgvector on Postgres), alongside metadata: source, date, version, and who is allowed to see it. That metadata matters more than people expect — it is what keeps one employee's HR document out of another team's answers.
Then, every question runs the same loop
When a question arrives, the system runs four steps in seconds:
- Embed the question with the same embedding model used for the documents. Mixing models here breaks the whole search, because the numbers would no longer be comparable.
- Retrieve candidates. The system finds the chunks closest in meaning to the question. A common pattern is to fetch 20–50 candidates, not just the top 5 — recall first, precision later.
- Rerank. A second, stronger model reads the question and each candidate together and re-scores them, cutting the list to the 5–10 passages that actually answer the question. This step catches near-misses: passages that sound similar but are about the wrong product or the wrong year.
- Augment and generate. The surviving passages are pasted into the prompt with an instruction like "answer using only these passages, and say so if they don't contain the answer." The language model then writes the answer, ideally with citations back to the sources.
One refinement is worth knowing: hybrid search. Pure meaning-based search is bad at exact strings — error codes, product SKUs, clause numbers. So production systems run a keyword search (an algorithm called BM25) in parallel and merge the two result lists. Meaning finds the concept; keywords find the exact thing.
A worked example
Back to the headphones. With RAG in place, the customer's question is embedded and searched against the store's help centre. The retriever pulls 30 chunks; the reranker keeps six.
Two of them are decisive: the returns-policy page ("Returns accepted within 14 days of delivery") and the exceptions page ("Headphones with broken seals are non-returnable"). The prompt contains both, plus the instruction to use only those passages.
The model now answers: "You have 14 days from delivery, and sealed headphones can't be returned once opened" — and it can link both pages. Same model, same question, completely different answer. Nothing about the model changed. What changed is what it was allowed to read.
RAG or fine-tuning?
Beginners often confuse the two, but they solve different problems. Fine-tuning — training a model further on your data — is for teaching behaviour: a consistent tone, a house style, a fixed output format. It is a poor way to teach facts, because facts change and the model will still guess when it is unsure.
RAG is for facts that live in documents, change often, or need to be auditable. If your question is "what changed last week?", RAG wins. If your question is "write every answer in this exact format?", fine-tuning wins. Many teams do both: fine-tune the behaviour, retrieve the facts.
What trips people up
Most RAG failures are retrieval failures, not generation failures. If the right passage never reaches the prompt, no prompt trick rescues the answer. The usual culprits are easy to list and hard to respect.
Stale documents. A RAG system answers from whatever is indexed. If the old 30-day policy is still in the index next to the new 14-day one, the model may quote either. Indexing is a maintenance job, not a one-time setup.
Bad chunking. Chunks that are too big flood the prompt with noise; chunks that are too small lose the sentence that made the passage meaningful. The orphan-chunk problem — a passage stripped of its heading — is the quietest killer of answer quality.
Permissions ignored. Retrieval must respect access control at the retrieval layer. If the search step ignores who is asking, the model will happily summarise documents the user was never meant to see.
Worth remembering: RAG reduces hallucinations, it does not eliminate them. As Wikipedia's RAG overview notes via Ars Technica, a model can still hallucinate around its source material — quoting a real passage and drawing a wrong conclusion from it. Citations are the safety net: they let a human check.
Questions people actually ask
Do I need a vector database to build RAG?
Not at first. For a few hundred documents, a simple in-memory index or Postgres with pgvector is plenty. Dedicated vector databases earn their place when you reach millions of chunks, heavy query traffic, or strict filtering needs.
Can RAG work with images, audio, or video?
Yes, with the same pattern: turn each item into an embedding with a multimodal model, store it, and retrieve by meaning. The generation step still works from text, so most systems retrieve a transcript, caption, or description alongside the media itself.
How do you know if your RAG system is good?
Keep a list of questions with known-good answers and check, for each one, whether the right passages were retrieved — before you even look at the generated answer. If the answer isn't in the retrieved set, the generator is not the problem. Fix retrieval first.
If you want to try this yourself, start small: take a folder of your own notes, chunk them, embed them with any free embedding model, and ask questions over them. The moment you see an answer arrive with a citation to a note you wrote months ago, the idea clicks. And the one thing to remember is simple: a RAG system is only as good as what it retrieves. Get retrieval right, and the generation mostly takes care of itself.
Sources: Lewis et al., "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks," NeurIPS 2020 (arXiv:2005.11401) · StackAI, RAG Explained · IBM, Retrieval-Augmented Generation pattern · Wikipedia, Retrieval-augmented generation
