How RAG actually works, part 10: choosing an embedding model in 2026

By · · AI · 5 min

Since re-embedding your whole corpus is the price of changing your mind: a practical, opinionated guide to picking an embedding model in 2026.

Part 10 of 10 in How RAG actually works, a hands-on tour of vectors, embeddings and retrieval.

Choosing an embedding model in 2026

Since re-embedding your whole corpus is the price of changing your mind, here's the short version so you choose once and choose well. Prices and specs are current as of mid-2026 and will drift, so verify before you commit real budget.

  • text-embedding-3-small (OpenAI) — 1,536 dimensions, ~$0.02 per million tokens, 8,191-token context. The safe default and the most widely deployed embedding model in production. Genuinely good, cheap enough that you can embed 50 million tokens for a dollar. If you're unsure, start here.
  • text-embedding-3-large (OpenAI) — 3,072 dimensions, ~$0.13 per million tokens. Better on harder retrieval (longer, more ambiguous, domain-specific). About 6× the cost; benchmark before you pay it.
  • Cohere embed-v4 — strong quality, a large context window and native text-plus-image embedding; pairs well with Cohere's reranker in one pipeline.
  • Voyage models — reach for these when retrieval quality is the bottleneck, and they're strong on code and technical docs specifically.
  • Open source (BGE-M3, Qwen3-Embedding, Nomic Embed, Jina v3) — the right call for confidential data, a minimal budget, or full control. BGE-M3 and Qwen3 are the open leaders in 2026, run them locally with Ollama for free, at the cost of operating your own GPU inference.

Two practical notes. Several models (OpenAI's included) support Matryoshka embeddings, which let you shorten a 3,072-dim vector to 1,024 or 256 dimensions with graceful quality loss, a real lever when storage or speed bites: going 3,072 → 256 is a ~12× cut in storage per vector. And if images or PDFs are on your roadmap in the next 6–12 months, starting with a multimodal model (Gemini Embedding, Voyage Multimodal, Cohere embed-v4) avoids a painful full re-embed later. Legacy note: OpenAI's old text-embedding-ada-002 is superseded, don't start new projects on it.

Storage sanity check, because it surprises people: a 1,536-dim float32 vector is about 6 KB, so a million chunks is roughly 6 GB of vectors before index overhead. Embedding costs are usually negligible next to that storage and next to your LLM inference bill.


Closing: it's simpler than it looks, and the work is in the details

Strip away the tooling and RAG is three moves. Turn text into coordinates that place meaning in space (embeddings). Find the coordinates nearest to a question (cosine similarity, made fast by a vector index). Hand those passages to an LLM and let it answer from them (generation). You built all three from scratch in part 4, in about sixty lines.

The frameworks and vector databases are conveniences over that core, and they're worth using at scale. But if the retrieval is bad, no framework saves you, and now you know where to look: your chunking, your embedding model, your metadata, and whether you've added hybrid search and reranking. The teams whose RAG feels magical aren't using secret models. They're the ones who understood these fundamentals and sweated the details.

Start with the toy version. Watch it retrieve a document that shares no keywords with the question. Once that clicks, the rest is engineering.


Found this useful? The best next step is to build the part 4 example yourself, swap in your own five documents, and ask it questions the documents answer in different words. The moment it retrieves the right chunk with zero keyword overlap is the moment embeddings stop being abstract.

FAQ

Which embedding model should I use if I'm not sure?

text-embedding-3-small from OpenAI. It's cheap, well-supported, and good enough for most retrieval tasks. Only move to a larger or specialized model once you've measured that retrieval quality is actually the bottleneck.

What happens if I switch embedding models later?

You have to re-embed your entire corpus. Vectors from two different models live in incompatible spaces, so mixing them produces meaningless distances. This is the main reason to choose deliberately upfront rather than defaulting to whatever a tutorial used.

Do I need a GPU to generate embeddings?

Not for hosted models like OpenAI's or Cohere's, those run as an API call. A GPU only matters if you choose to self-host an open-source embedding model like BGE-M3 or Qwen3-Embedding, typically for cost or data-residency reasons.

How much does embedding a large document set cost?

Very little relative to LLM inference. text-embedding-3-small costs roughly $0.02 per million tokens, so embedding 50 million tokens costs about a dollar. Storage and your LLM's per-query cost dominate the budget, not embedding.

Related reading