RAG and vector search, explained for managers

By · · AI · 7 min

Your team wants to build an 'AI search' feature and the proposal is full of terms like embeddings and vector databases. Here's what's actually happening under the hood, one worked example, and the questions worth asking before you approve the budget.

Every vendor pitch this year seems to include the phrase "we'll use RAG." It's usually said with confidence, and it usually goes unquestioned, because pushing back means admitting you don't know what RAG stands for. You don't need to write one to approve or reject the budget for one. You do need to understand what it buys you, what it costs, and what can still go wrong.

The problem RAG actually solves

A general-purpose AI model (the kind behind ChatGPT, Claude, or Gemini) is trained on a huge slice of the public internet. It's fluent, but it knows nothing about your company: your refund policy, your product catalogue, last quarter's incident postmortems. Ask it a question about your own business and it will either say "I don't know" or, worse, confidently make something up.

There are two ways to fix that:

  1. Fine-tune the model on your company's documents. Expensive, slow, and the moment a policy document changes, the model's knowledge is out of date again until you retrain it.
  2. Retrieval-Augmented Generation (RAG): don't touch the model at all. Instead, before asking it a question, go fetch the handful of company documents most relevant to that question, and hand them to the model as context along with the question.

RAG is popular because it's the second option: cheap, fast to update (add a document, it's searchable within minutes), and it lets the model cite where an answer came from. The cost is that the whole system is only as good as its retrieval step, which is where vector indexing comes in.

What a vector index actually is

Retrieval needs a way to find "the paragraphs relevant to this question" out of possibly hundreds of thousands of paragraphs. Old-school keyword search (matching the literal words) breaks down fast: a customer asking "can I get my money back" won't match a policy document that only ever says "refund eligibility."

A vector index solves this by converting every chunk of text into a list of a few hundred to a few thousand numbers, called an embedding, produced by a model trained to place similar meanings near each other in that number space. "Can I get my money back" and "refund eligibility" end up as two points sitting close together, even though they don't share a single word. Search stops being about matching text and becomes about measuring distance between points.

That's it. That's the whole trick. Everything else is engineering around that one idea.

A 2D map of document chunks clustered by meaning, with an incoming question landing next to the three chunks it matches and a dashed radius showing which chunks get retrieved

Example

Say your company has a 400-page HR handbook, and you want an internal chatbot that answers employee questions about it correctly, instead of pointing people to a PDF nobody opens.

Here's what happens end to end:

  1. Chunking. The handbook is split into roughly 1,200 overlapping paragraphs (a whole document is too big and too unfocused to hand to the model in one go).
  2. Embedding. Each of those 1,200 chunks is run through an embedding model, producing a vector for each one. These vectors get stored in a vector database alongside the original text.
  3. A question comes in. An employee types: "How many weeks of parental leave do I get?"
  4. The question is embedded too, using the same embedding model, producing one vector for the question.
  5. Nearest-neighbour search. The database finds the handful of stored vectors closest to the question's vector, typically the 3 to 8 most relevant chunks. This takes tens of milliseconds even across a huge index.
  6. Generation. Those chunks (the actual paragraphs about parental leave, verbatim) get pasted into the prompt sent to the language model, along with an instruction: "answer using only the following context." The model produces a fluent answer, ideally with a reference back to the source section.

A six-step pipeline diagram: a question is embedded, matched against a vector index, the closest chunks are retrieved, and handed to the model alongside the question to produce a grounded answer

Notice what the language model never did: it never "learned" your HR policy. It was handed the relevant three paragraphs and asked to summarise them. If HR updates the parental leave policy tomorrow, you re-embed that one paragraph, and the chatbot is instantly current, no retraining involved.

Where this goes wrong, and what to ask

Most RAG failures aren't the language model's fault, they're retrieval failures further upstream. When you're reviewing a proposal, these are the questions worth asking:

  • "What happens when a source document changes or gets deleted?" If there's no answer, stale or contradictory information will surface indefinitely.
  • "How do you chunk long documents?" Splitting badly (mid-sentence, mid-table) is the single most common cause of an otherwise-good system giving confidently wrong answers.
  • "Does the answer show its sources?" If the chatbot can't point to which document it used, you have no way to audit a wrong answer, and neither does anyone else.
  • "What's the cost model?" Pricing is usually a combination of embedding cost (per document, mostly one-off), storage, and a per-query language model call. The last one scales with usage, so a wildly popular internal tool can get expensive quietly.
  • "What happens with a question that has no good match in the documents?" A well-built system says "I couldn't find anything relevant." A poorly-built one still generates a fluent, wrong answer, because nothing stops the model from filling gaps with confidence.

When RAG isn't the right tool

RAG is built for questions with answers hiding in text: policies, documentation, support tickets, contracts. It is a poor fit for questions that require calculation over structured data, "what was our refund rate by region last quarter" is a database query and a dashboard, not a document to retrieve. If a proposal reaches for RAG to answer questions that are really aggregations over a database, that's a sign the team reached for the trendy tool instead of the appropriate one.

The underlying idea, though, holds up regardless of the hype cycle: don't retrain a general model on your private data, retrieve the relevant slice of it on demand and let the model reason over what's actually in front of it. That's a sound engineering decision independent of whether "RAG" is this year's buzzword or next year's assumed default.

FAQ

What is RAG in simple terms?

RAG (Retrieval-Augmented Generation) is a way of answering questions using an AI model by first fetching the few most relevant paragraphs from your own documents, then asking the model to answer using only those paragraphs. It's the difference between an AI guessing from general knowledge and an AI reading the actual policy before answering.

Is RAG expensive to build and run?

It's usually the cheapest of the realistic options. Compared to fine-tuning a model on your data, RAG needs no retraining when documents change, and the ongoing cost is mostly a small embedding fee per document plus a per-query model call, both far smaller than most teams expect.

How is RAG different from just using ChatGPT or Claude directly?

A general model like ChatGPT or Claude knows nothing about your company's private documents. RAG is the layer that finds your company's relevant paragraphs and hands them to the model as context, so the answer is grounded in your actual policies instead of the model's general training data.

What's the biggest risk when a vendor proposes a RAG project?

That retrieval quietly fails and nothing tells you. If the system can't find the right paragraph, a poorly built one still produces a fluent, confident, wrong answer instead of saying "I don't know." Ask for source citations on every answer, that's the cheapest way to catch this before customers do.

For developers: where do I start building one?

See How RAG actually works, part 1, a hands-on developer series that builds a working RAG system from scratch in part 4.

Related reading