How RAG actually works, part 1: the problem RAG solves
By Guus van de Wal · · AI · 3 min
Why fine-tuning and prompt-stuffing both fall short, and why splitting the problem into retrieval plus generation is what makes RAG work at all.
If you've built anything with an LLM in the last two years, you've heard "just use RAG." And you probably did: you dropped in a framework, pointed it at some PDFs, and it worked. Sort of. Then it started returning the wrong chunks, hallucinating around the gaps, and you had no idea which knob to turn.
This article is the thing I wish I'd read before that happened. We'll build a tiny retrieval system from scratch in TypeScript so the mechanics are impossible to hide, then scale the same ideas up to something production-shaped. No framework magic. By the end you'll know what an embedding is, why cosine similarity is the whole game, where chunking quietly ruins everything, and how to pick tools without cargo-culting a blog post.
The problem RAG solves
An LLM knows an enormous amount, but it doesn't know your things: your product docs, last week's support tickets, the internal wiki, the contract a customer signed in 2023. It also has a fixed knowledge cutoff and a limited context window, and it will confidently make things up when asked about something it never saw.
You have three options for feeding it your knowledge:
- Fine-tune the model on your data. Expensive, slow to iterate, and it bakes knowledge in as fuzzy weights rather than retrievable facts. Update a document and you have to retrain. Great for teaching style or format, poor for teaching facts that change.
- Stuff everything into the prompt. Simple, but you hit the context window fast, you pay for every token on every call, and models get measurably worse at finding the needle when the haystack is huge.
- Retrieve just the relevant pieces at query time and put those in the prompt. This is RAG: Retrieval-Augmented Generation.
RAG works because it splits the problem in two. A retrieval system finds the handful of passages most likely to answer the question, and the generation step (the LLM) reads only those passages and writes the answer. The model stops being a memory and becomes a reasoning engine over text you hand it. Update a document, re-index it, and the next answer is current. No retraining.
The magic that makes retrieval work, matching a question to relevant text even when they share no keywords, is embeddings. So let's start there.
In this series
This is a 10-part, hands-on tour of RAG for developers: everything is built from scratch first, then scaled to something production-shaped.
- the problem RAG solves (you are here)
- what an embedding actually is
- cosine similarity, the heart of it all
- build a tiny RAG from scratch
- chunking, where most RAG systems quietly fail
- going to production with a vector database
- making retrieval actually good
- what this looks like in the real world
- common ways RAG goes wrong
- choosing an embedding model in 2026
FAQ
What does RAG stand for?
Retrieval-Augmented Generation. It's a pattern where a language model answers a question using passages retrieved from your own documents at query time, instead of relying only on what it learned during training.
Is RAG the same as fine-tuning?
No. Fine-tuning changes a model's weights to bias its behavior or style, and it doesn't reliably store retrievable facts. RAG leaves the model untouched and instead retrieves the relevant text at query time, so updating a document takes effect immediately with no retraining.
Do I need a vector database to build RAG?
Not to start. Part 4 builds a complete, working RAG system with a plain array and a loop. A vector database becomes worth adding once a linear scan over your documents is too slow, see part 6.
Does RAG eliminate hallucination?
It reduces it substantially by grounding answers in retrieved text, but it doesn't eliminate it. A model can still misread or over-generalize from the passages it's given. See part 9 for the specific failure modes and how to catch them.