How RAG actually works, part 5: chunking, where most RAG systems quietly fail
By Guus van de Wal · · AI · 4 min
Chunking is the highest-leverage, easiest-to-get-wrong decision in any RAG system. Here's how documents actually get split, and why it quietly determines everything downstream.
Part 5 of 10 in How RAG actually works, a hands-on tour of vectors, embeddings and retrieval.
Chunking, where most RAG systems quietly fail
Here's the part the tutorials skip and the reason your RAG returns garbage. You rarely embed whole documents. A 40-page handbook becomes one vector that means "vaguely about everything," which matches nothing well. So you split documents into chunks and embed each chunk. How you split is one of the highest-leverage decisions in the whole system, and it's easy to get wrong.
Chunk too large and each vector blurs several topics together; retrieval gets imprecise and you waste context-window space on irrelevant text. Chunk too small and you sever the context a passage needs to make sense; a chunk reading "It must be filed within 14 days" is useless if the it lived in the previous chunk.
A few strategies, roughly in order of sophistication:
- Fixed-size chunks (e.g. 500 tokens) are simple and predictable but slice mid-sentence and mid-idea. Rarely good enough alone.
- Overlapping windows carry the last ~10–15% of one chunk into the start of the next, so a thought split across a boundary survives in at least one chunk. Cheap and effective; a sane default.
- Structural chunking splits on the document's own seams, paragraphs, Markdown headings, sections, so each chunk is a coherent unit. Usually the best bang for the buck when your documents have structure.
- Semantic chunking uses embeddings to detect where the topic shifts and cuts there. Slower to build, but keeps ideas whole.
A pragmatic recipe that works for most text: split on headings and paragraphs first, then pack those into ~300–800 token chunks with ~15% overlap, and, critically, attach metadata to every chunk (source document, title, section, URL, date). That metadata earns its keep later for filtering and for showing citations.
interface Chunk {
id: string;
text: string;
embedding: number[];
metadata: {
source: string;
title: string;
section?: string;
url?: string;
updatedAt: string;
};
}
One rule to tattoo on your monitor: the query and the chunks must be embedded with the same model. Vectors from two different models live in incompatible spaces and their distances are meaningless. And if you ever switch embedding models, you have to re-embed your entire corpus, there's no shortcut, so choose deliberately (more on that below).