How RAG actually works, part 7: making retrieval actually good
By Guus van de Wal · · AI · 4 min
Getting RAG running is a weekend. Getting retrieval good is the real work: hybrid search, reranking, metadata filtering, query rewriting, and how to measure it.
Part 7 of 10 in How RAG actually works, a hands-on tour of vectors, embeddings and retrieval.
Making retrieval actually good
Getting RAG running is a weekend. Getting retrieval good is the real work, because the LLM can only be as accurate as the chunks you feed it. If the right passage never surfaces, no amount of prompt engineering saves the answer. A few techniques that move the needle, in rough order of impact:
Hybrid search. Pure vector search is brilliant at meaning but can whiff on exact tokens, product SKUs, error codes, names, acronyms, where the literal string matters. Classic keyword search (BM25) nails those but misses paraphrases. Hybrid search runs both and fuses the scores, and it consistently beats either alone. Weaviate, Qdrant and Pinecone offer it natively; with pgvector you combine vector distance and Postgres full-text search in SQL yourself, which is the most powerful and the most manual option.
Reranking. Retrieve a generous first pass (say the top 30 by vector similarity), then run a dedicated cross-encoder reranker that reads the query and each candidate together and scores relevance far more precisely than a raw embedding comparison. Keep the top 5 of those. It adds latency and cost, but it's one of the biggest single quality jumps available, especially with ambiguous queries.
Metadata filtering. As shown above, constrain the search to the relevant subset (tenant, language, date, document type) before or during the vector search. Faster and more accurate, since you're not ranking against irrelevant candidates.
Query rewriting. Real user questions are messy ("it's broken again"). Cheaply have an LLM expand or rephrase the query, or generate a hypothetical ideal answer and embed that (a trick called HyDE), before retrieving. Often a small change with outsized returns on vague inputs.
And measure. You cannot improve retrieval you don't evaluate. Build a small test set of real questions paired with the chunks that should be retrieved, and track metrics like recall@k (did the right chunk appear in the top k?) as you tune. Everything downstream, the LLM's answer, its citations, its factual accuracy, depends entirely on whether the right chunks surfaced in the first place.