How RAG actually works, part 9: common ways RAG goes wrong

By · · AI · 3 min

The predictable ways RAG systems fail, from bad chunking to silent embedding-model mismatches, and how to catch each one before users do.

Part 9 of 10 in How RAG actually works, a hands-on tour of vectors, embeddings and retrieval.

Common ways RAG goes wrong

The failure modes are predictable once you know them:

  • Retrieval returns the wrong chunks. Almost always a chunking or embedding-model problem, not an LLM problem. Fix retrieval before you touch the prompt. If the right chunk isn't in the top results, the model never had a chance.
  • Chunks too big or too small. Too big blurs meaning; too small severs context. Revisit chunk size and overlap early; it's the cheapest large win.
  • The answer is in the docs but never retrieved. Add hybrid search (for exact terms), reranking (for precision), or query rewriting (for vague questions).
  • Model hallucinates around gaps. Instruct it explicitly to answer only from the provided context and to say "I don't know" otherwise, then show its citations so users can verify. RAG reduces hallucination; it doesn't abolish it.
  • Stale index. Embeddings are a snapshot. If your docs change, you need a re-indexing pipeline (on a schedule, or triggered by document updates). A RAG system quietly drifts out of date without one.
  • Silent embedding-model mismatch. Query embedded with one model, corpus with another → meaningless distances → nonsense results, with no error thrown. Pin the model everywhere.
  • No evaluation. If you're tuning by vibes, you're guessing. A small labelled test set and a recall@k number turns guessing into engineering.

Related reading