How RAG actually works, part 6: going to production with a vector database
By Guus van de Wal · · AI · 5 min
Moving from a linear scan to a real vector index: HNSW, pgvector in practice, and an honest, opinionated tour of the 2026 vector database landscape.
Part 6 of 10 in How RAG actually works, a hands-on tour of vectors, embeddings and retrieval.
Going to production with a vector database
Our from-scratch version loops over every chunk on every query. At five documents, instant. At five million, hopeless: that's a linear scan computing millions of dot products per request. Vector databases exist to make "find the nearest vectors" fast, using an approximate nearest neighbour (ANN) index, most commonly HNSW (Hierarchical Navigable Small World graphs). ANN trades a sliver of accuracy for enormous speed, turning a full scan into something that touches a tiny fraction of the data. In practice a well-built index searches a million vectors in around ten milliseconds.
A vector database also handles the operational reality our toy skips: persistence, metadata filtering, concurrent reads and writes, incremental updates, and backups.
If you already run Postgres, the path of least resistance is pgvector, an extension that adds a vector column type and similarity operators to a database you already operate. No new infrastructure, no new bill, and your embeddings live next to your relational data so you can filter with plain SQL.
-- Enable the extension
CREATE EXTENSION IF NOT EXISTS vector;
-- A table with a 1,536-dim vector column (matches text-embedding-3-small)
CREATE TABLE chunks (
id BIGSERIAL PRIMARY KEY,
content TEXT NOT NULL,
source TEXT NOT NULL,
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
embedding VECTOR(1536)
);
-- An HNSW index for fast approximate search using cosine distance
CREATE INDEX ON chunks
USING hnsw (embedding vector_cosine_ops);
-- Retrieve the 5 nearest chunks to a query vector,
-- filtered to recent docs from one source. `<=>` is cosine distance.
SELECT content, source, 1 - (embedding <=> $1) AS similarity
FROM chunks
WHERE source = 'handbook' AND updated_at > now() - interval '1 year'
ORDER BY embedding <=> $1
LIMIT 5;
That WHERE clause is the quiet superpower of putting vectors in your main database: metadata filtering. Real production queries are almost never "find the 5 nearest chunks." They're "find the 5 nearest chunks from this customer's documents, in English, updated since January." Being able to combine semantic search with ordinary filters, in one query, is worth a lot.
The 2026 landscape, briefly and honestly
The field has consolidated. You don't need to agonise; you need to match the tool to your scale and your existing stack:
- pgvector — you already run Postgres and are under ~tens of millions of vectors. Production-grade in 2026; companies like Supabase, Neon and Instacart run it at real scale. The ceiling is single-node Postgres limits, not the extension. Start here if it applies to you.
- Chroma — prototyping and local development. Zero-config, the default in most framework tutorials. Plan your migration path before production; the good news is that switching backends is usually a few lines through LangChain or LlamaIndex.
- Pinecone — you want fully managed and would rather not operate vector infrastructure. The serverless model fixed the old cost complaints; it's the "safe default" for zero-ops teams willing to pay for calm.
- Qdrant — open-source, Rust-built, the latency leader among self-hosted options. Strong filtering and quantization. Pick it when speed dominates and you're happy to self-host.
- Weaviate — open-source with the richest built-in hybrid search and even modules that vectorize text for you on insert.
- Milvus / Vespa — reach for these at the billion-vector, distributed-deployment tier. Overkill below that.
Don't pick on headline benchmarks. Pick on: managed vs. self-hosted, your scale tier, whether you need hybrid search, and what data platform you already run. At the same recall level, all of these produce equivalent RAG answer quality; the difference is how hard you work to hit that recall at your scale.