Retrieval-Augmented Generation: Embeddings, Vector Search, and Why It Beats a Bigger Prompt

The Problem RAG Actually Solves

The previous lesson in this series ended on a specific problem: a language model has no database lookup and no fact-checking step built in, so it generates the statistically likely continuation of your prompt whether or not that continuation is true. That's not a bug a bigger or newer model fixes - a larger model just makes the fluent, confident wrong answer more convincing, not less likely. Retrieval-augmented generation, RAG, doesn't make the model smarter. It changes what the model is being asked to do: instead of recalling a fact from training, it summarizes real documents you hand it directly in the prompt, retrieved fresh for that specific question.

Why Not Just Paste Everything Into the Prompt?

If retrieval is the fix, the obvious shortcut is skipping it entirely - why not paste your whole knowledge base into every prompt and let the model sort it out? Two practical limits rule this out. Every model has a context window, a hard cap on how many tokens it can read per call, and a real knowledge base blows past that within a handful of documents. Even for the ones that would technically fit, every token costs money and adds latency on every request, whether or not it's relevant to the question being asked. There's a subtler cost too: the 'instructions buried in the middle get ignored' failure mode from the previous lesson applies just as much to shoved-in documents as to shoved-in rules - a model handed 40 pages of mostly-irrelevant context doesn't read all of it evenly, and is more likely to miss the one paragraph that actually answers the question. Retrieval exists to do that filtering ahead of time, cheaply, before the model sees anything at all.

What an Embedding Actually Is

An embedding is what an embedding model produces from a piece of text: a fixed-length list of floating point numbers, typically somewhere between 384 and 3072 of them depending on the model. The list itself isn't meant to be read by a person - no single number corresponds to a concept you could name. What matters is the list's position in that numeric space relative to other texts' embeddings: the model is trained so that texts with similar meaning land close together and texts with different meaning land far apart, regardless of which exact words either one uses. A chunk of text becomes a point in space, and "find related content" becomes "find nearby points."

{
  "input": "How do I reset my password?",
  "model": "text-embedding-3-small",
  "embedding": [0.0182, -0.0431, 0.0927, -0.0058, 0.1140, "... 1531 more numbers"],
  "dimensions": 1536
}

Why "Semantic" Beats a Keyword Match

A traditional keyword search matches on shared words and their variants - fast and precise for exact terms, but blind to meaning. Ask it about "resetting my password" and it will happily surface a document about signup-form validation rules that happens to contain the word "password," while missing a perfectly relevant page titled "recovering account access" that never uses that word at all. A vector search compares meaning instead of matching words: it embeds the query the same way it embedded every stored chunk, then finds the chunks whose embeddings sit closest to the query's embedding in that numeric space.

Measuring Closeness: Cosine Similarity

"Closest" needs an actual definition, and the standard one for embeddings is cosine similarity - it measures the angle between two vectors rather than their raw distance, so it isn't thrown off by one piece of text simply being longer, and its vector correspondingly larger in magnitude, than another. A cosine similarity of 1.0 means the two vectors point in exactly the same direction, as close to identical meaning as the model can express; 0 means unrelated; -1 means opposite. This is the actual arithmetic underneath every "semantic search," run at whatever scale the corpus requires.

def cosine_similarity(a: list[float], b: list[float]) -> float:
    dot = sum(x * y for x, y in zip(a, b))
    norm_a = sum(x * x for x in a) ** 0.5
    norm_b = sum(y * y for y in b) ** 0.5
    return dot / (norm_a * norm_b)

At real scale, a vector database doesn't run this comparison against every stored vector one at a time - checking a query against a million stored vectors linearly would be far too slow for something that has to happen inline on every request. It builds an approximate nearest-neighbor index instead (HNSW is the algorithm most popular vector databases use), which finds very-likely-nearest matches in a fraction of the time, trading a small amount of exactness for a large amount of speed. The comparison it's approximating, though, is still the cosine similarity above.

Vector Databases: What They're Actually Storing

Strip away the marketing and a vector database does three specific jobs: store a vector alongside the original text and any metadata (source filename, page number, timestamp), index those vectors for fast approximate nearest-neighbor search, and answer a query of the form "give me the k closest vectors to this one." That's the whole interface. Purpose-built options exist (Pinecone, Weaviate, Qdrant, Chroma), and so does bolting vector search onto a database you're already running - Postgres's `pgvector` extension is a common choice specifically because it means one less system to operate, at the cost of not being as purpose-tuned for very large vector workloads as a dedicated vector database.

Chunking: The Step Before Embedding

You don't embed a whole document as one vector - a 50-page PDF compressed into a single point loses almost all of its specific detail. You split it into chunks first, embed each chunk separately, and store them as separate entries. Chunk size is a real tradeoff, not a default to leave alone: chunks too small lose surrounding context (a sentence about "the fee" with no indication of which fee), chunks too large dilute the embedding's precision (a chunk covering five different subtopics produces a vector that's a blurry average of all five, close to none of them). A small overlap between consecutive chunks - so the last few sentences of one chunk repeat as the first few of the next - helps stop a single fact from being awkwardly split right at a chunk boundary.

import glob
from pathlib import Path

CHUNK_SIZE = 500     # words per chunk
CHUNK_OVERLAP = 50   # words repeated between consecutive chunks


def chunk(text: str, size: int = CHUNK_SIZE, overlap: int = CHUNK_OVERLAP) -> list[str]:
    words = text.split()
    step = size - overlap
    return [" ".join(words[i:i + size]) for i in range(0, len(words), step)]


def embed(text: str) -> list[float]:
    response = embedding_client.embed(model="text-embedding-3-small", input=text)
    return response.data[0].embedding


def ingest(corpus_dir: str, collection) -> int:
    ingested = 0
    for path in sorted(glob.glob(f"{corpus_dir}/*")):
        text = Path(path).read_text()
        for i, piece in enumerate(chunk(text)):
            collection.add(
                ids=[f"{Path(path).name}-{i}"],
                embeddings=[embed(piece)],
                documents=[piece],
                metadatas=[{"source": Path(path).name}],
            )
        ingested += 1
    return ingested

The RAG Loop at Query Time

Ingestion happens ahead of time, and only when source documents change. Retrieval happens on every single request, inline, in the critical path of answering a question - which is exactly why it needs to be fast. Four steps, every time: embed the incoming question with the same embedding model used during ingestion (mixing models here silently breaks everything, since their vector spaces aren't compatible with each other), search the vector store for the top-k closest chunks, paste those chunks into the prompt as context, and send the whole thing to the LLM.

def answer_with_rag(question: str, collection, chat_client) -> str:
    query_embedding = embed(question)
    results = collection.query(query_embeddings=[query_embedding], n_results=5)
    context = "\n\n---\n\n".join(results["documents"][0])

    response = chat_client.chat(messages=[
        {
            "role": "system",
            "content": "Answer using only the context below. If the answer isn't in it, say so - do not guess.",
        },
        {"role": "system", "content": f"Context:\n{context}"},
        {"role": "user", "content": question},
    ])

    return response.content

Failure Mode: Naive Top-k Isn't Enough

The obvious version of retrieval - embed the query, grab the top 3 or top 5 closest chunks, done - works well on straightforward questions, and fails in a specific, recurring way on harder ones: the genuinely relevant chunk exists in the corpus, but it's ranked 7th or 12th instead of inside the top handful, so it never reaches the prompt at all. This isn't rare. One real internal evaluation against a genuine multi-document knowledge base found that some question phrasings needed the vector search to return the top 20 to 50 candidates before the correct source document even appeared in the results - a plain top-3 search would have missed it completely. Just widening top-k "to be safe" has its own cost: handing the model 40 loosely-related chunks instead of 3 tightly-related ones reintroduces the exact noise-in-a-long-prompt problem retrieval was supposed to solve in the first place.

Retrieve Broad, Then Rerank

The fix is a second, more expensive scoring step, applied only to a widened shortlist rather than the whole corpus: retrieve broadly with the fast vector search (top 20-50, cast a wide net), rerank that shortlist with a cross-encoder model, and keep only the true top few after reranking. A cross-encoder is a different kind of model from an embedding model - instead of computing two vectors independently and comparing them afterward, it takes the query and a candidate chunk together as a single input and scores that pair directly. That joint view lets it weigh specific interactions between the query's exact wording and the chunk's exact content, which is far more accurate than comparing two embeddings computed with no knowledge of each other. It's also much slower per comparison, which is exactly why it only ever runs on the already-narrowed shortlist, never the full corpus.

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("BAAI/bge-reranker-v2-m3")


def rerank(query: str, candidates: list[str], top_n: int) -> list[tuple[str, float]]:
    pairs = [(query, candidate) for candidate in candidates]
    scores = reranker.predict(pairs)  # scores each (query, chunk) pair jointly

    ranked = sorted(zip(candidates, scores), key=lambda pair: pair[1], reverse=True)
    return ranked[:top_n]

Measuring Whether Retrieval Actually Works

None of the above is worth trusting on faith. The way to actually know if a retrieval setup works is the same way you'd check any other engineering change: build a small labeled test set - a list of realistic questions, each paired with the document that should answer it - and check, for every question, whether that expected document actually shows up in the retrieved results. Run it again after any change to chunk size, embedding model, or reranking, and you have a real number to compare instead of a feeling. One real evaluation run this way, against a genuine multi-format internal knowledge base of dozens of documents, found the expected source document in the top 3 retrieved chunks for 8 out of 10 test questions - and both misses turned out to be adjacent, related documents pulled in by genuine semantic overlap, not a broken parser or a bug, which is a very different and much less worrying failure than it would otherwise have been.

RAG Isn't Automatically Better

It's tempting to treat "add RAG" as a strict upgrade, and the honest data doesn't support that. One real before-and-after comparison - same model, same 30 real questions, with the only variable being whether retrieved context was injected into the prompt - found RAG fixed the single most damaging failure mode, the model confidently fabricating an answer instead of admitting it didn't know, in several cases. But it also made a handful of previously-correct answers worse, because the extra retrieval step ate into the model's available reasoning turns and occasionally pointed it at a plausible-but-wrong document instead of the right one. Net effect: a real improvement on the highest-stakes failure mode, not a strict win on every single question.

Where This Actually Matters

None of this is about making a demo look impressive. Every piece above maps onto a real decision you make once retrieval is powering an actual feature: what chunk size and overlap to use, which embedding model to standardize on (and never silently change without re-embedding everything that came before it), whether pgvector is enough or a dedicated vector database is worth the extra operational cost, whether a reranking pass is worth its added latency for your particular corpus, and how you'll know - with a number, not a feeling - whether any of it actually improved answers. Retrieval-augmented generation isn't one single technique so much as this whole stack of small, testable engineering decisions, each with a real tradeoff behind it.