How RAG actually works, part 4: build a tiny RAG from scratch

By · · AI · 6 min

A complete, working retrieval system in under sixty lines of TypeScript: no framework, no vector database, just embeddings, a loop, and an LLM call.

Part 4 of 10 in How RAG actually works, a hands-on tour of vectors, embeddings and retrieval.

Build a tiny RAG from scratch

Let's assemble a complete, working retrieval system with zero libraries beyond the OpenAI SDK. No vector database, no framework. It stores everything in a plain array and searches by looping. This is genuinely how every RAG system works; the fancy tools just make this loop fast at scale.

import OpenAI from "openai";

const openai = new OpenAI();

// ---- 1. Our "knowledge base": a few documents ----
const documents = [
  "To reset your password, click 'Forgot password' on the login screen and follow the emailed link.",
  "Our refund policy allows returns within 30 days of purchase with a valid receipt.",
  "The mobile app supports iOS 16 and above, and Android 11 and above.",
  "Business-plan customers get priority support with a 2-hour response-time guarantee.",
  "To cancel your subscription, go to Settings → Billing → Cancel plan. Access continues until the period ends.",
];

// ---- 2. Embed every document once, up front (this is 'indexing') ----
interface Chunk {
  text: string;
  embedding: number[];
}

async function embedAll(texts: string[]): Promise<number[][]> {
  const res = await openai.embeddings.create({
    model: "text-embedding-3-small",
    input: texts, // the API accepts a batch — one call, many vectors
  });
  return res.data.map((d) => d.embedding);
}

const embeddings = await embedAll(documents);
const index: Chunk[] = documents.map((text, i) => ({
  text,
  embedding: embeddings[i],
}));

// ---- 3. Retrieval: embed the query, score every chunk, take the top-k ----
function cosineSimilarity(a: number[], b: number[]): number {
  let dot = 0,
    na = 0,
    nb = 0;
  for (let i = 0; i < a.length; i++) {
    dot += a[i] * b[i];
    na += a[i] * a[i];
    nb += b[i] * b[i];
  }
  return dot / (Math.sqrt(na) * Math.sqrt(nb));
}

async function retrieve(query: string, k = 2): Promise<Chunk[]> {
  const [queryEmbedding] = await embedAll([query]);
  return index
    .map((chunk) => ({
      chunk,
      score: cosineSimilarity(queryEmbedding, chunk.embedding),
    }))
    .sort((a, b) => b.score - a.score)
    .slice(0, k)
    .map((r) => r.chunk);
}

// ---- 4. Generation: hand the retrieved chunks to the LLM ----
async function answer(query: string): Promise<string> {
  const context = await retrieve(query);
  const contextText = context.map((c, i) => `[${i + 1}] ${c.text}`).join("\n");

  const res = await openai.chat.completions.create({
    model: "gpt-4o-mini",
    messages: [
      {
        role: "system",
        content:
          "Answer the question using ONLY the context below. " +
          "If the context doesn't contain the answer, say you don't know. " +
          "Cite the source number you used.\n\n" +
          `Context:\n${contextText}`,
      },
      { role: "user", content: query },
    ],
  });
  return res.choices[0].message.content ?? "";
}

// ---- Try it ----
console.log(await answer("How do I stop paying for my plan?"));

Run it and ask "How do I stop paying for my plan?" Notice the question contains none of the words "cancel," "subscription," or "billing", yet the retriever surfaces the cancellation document, because their meanings are close in embedding space. The LLM then answers from that chunk and cites it. That's the full RAG loop:

index once (embed your docs) → retrieve per query (embed the question, find nearest chunks) → generate (LLM answers from those chunks).

Everything from here is making each of those three steps better, faster, or more accurate.

FAQ

Do I need LangChain or LlamaIndex to build RAG?

No. The example above is a complete, working RAG loop in under sixty lines using only the OpenAI SDK. Frameworks add convenience once you're managing many document sources, chunking strategies, or retrievers, but they wrap the same three steps: embed, retrieve, generate.

Why does the retriever find the right document with no keyword overlap?

Because retrieval works on embeddings, not literal text. See part 2 and part 3 for why semantically similar text ends up close together in embedding space regardless of shared vocabulary.

How do I make this scale past five documents?

Swap the in-memory array and loop for a real vector index once a linear scan becomes too slow, see part 6 for pgvector and the 2026 vector database landscape.

What model should I use for the embedding step?

text-embedding-3-small is a safe, cheap default for most use cases. Part 10 covers the full tradeoffs between OpenAI, Cohere, Voyage, and open-source options.

Related reading