A RAG pipeline retrieves the top-10 chunks by cosine similarity, feeds them to the LLM, and the team ships it. The embedding model scores a chunk at 0.87 — clearly relevant, right? The LLM generates a confident answer. The demo looks good.
Six weeks later, users start reporting hallucinations. Not because the LLM is making things up from nothing — it's faithfully reading the context it was given. The context is just wrong. The retrieval step is surfacing documents that are semantically close to the query but don't actually answer it. And nobody measured this, because they were watching cosine similarity, which told them nothing useful.
The semantic gap: what cosine similarity actually measures
Cosine similarity between two embedding vectors measures the angle between them in high-dimensional space. Embeddings are trained to project semantically related text nearby. "How do I reset my password?" and "Password reset instructions for enterprise users" land close together. So do "What is our refund policy?" and "Our refund policy was updated in Q3 2024 to include digital goods." That second pair looks great for a question about refund policy — and it would be, if the user asked a general question. But if they asked "Can I get a refund on a software subscription purchased before Q3 2024?" the Q3 update document might be the worst possible context to retrieve.
The fundamental problem: cosine similarity measures topical overlap, not answer relevance. A document about the same topic as the query will score high even if it answers a different question entirely, addresses a different time period, or contains information that actively contradicts what the user needs.
False positives: high similarity, wrong context
Query: "What is the maximum file size for API uploads?"
Retrieved chunk (cosine sim: 0.91): "API upload limits vary by plan. Enterprise customers can upload files up to 5 GB. See our pricing page for details."
Similarity is high — the document is clearly about API upload limits. But the user is on the free tier and this chunk never mentions the free tier limit (which is 25 MB). The LLM reads the context, finds no free tier answer, and either hallucinates or says "it depends on your plan" — neither of which helps.
False negatives: low similarity, right context
Query: "The export button is grayed out. Why?"
Skipped chunk (cosine sim: 0.61): "Feature availability: Export to CSV requires a Business or Enterprise subscription. Users on Free or Starter plans will see export controls disabled."
The user described a UI symptom. The relevant document describes a subscription gate. The embedding model doesn't know these two sentences belong together — "grayed out" and "controls disabled" are related in product context but not in the embedding space trained on general text. The right chunk scores below the retrieval threshold and never makes it into the context window.
Measuring retrieval correctly: the four metrics that matter
Proper RAG retrieval evaluation requires a labeled dataset: a set of queries, each paired with the ground-truth set of relevant document chunks. Building this dataset is the hard part — it requires either human annotation or a synthetic generation pipeline where you generate questions from your documents and verify the answers. Without it, you are flying blind.
With it, you can compute the metrics that actually characterize retrieval quality.
Precision@K
Of the K chunks returned, what fraction are actually relevant?
Precision@K = |relevant chunks in top-K| / KExample: you retrieve K=5 chunks for a query. Your labeled dataset says 2 of them are truly relevant. Precision@5 = 2/5 = 0.40. You're filling 60% of your context window with noise.
Precision@K is the metric that directly captures the "context stuffing" failure mode. If your LLM context window has 10 chunks and Precision@10 is 0.20, you've given the model 8 irrelevant documents to wade through. The model's attention is diluted across junk. This degrades generation quality even when the 2 relevant chunks are present — a finding that has been replicated across multiple studies of long-context LLM behavior.
Recall@K
Of all relevant chunks for this query, how many did you retrieve in your top K?
Recall@K = |relevant chunks in top-K| / |total relevant chunks|Example: there are 4 relevant chunks for a query in your corpus. With K=5, you retrieved 3 of them. Recall@5 = 3/4 = 0.75. You're missing 25% of the information the model needs to answer correctly.
Recall@K and Precision@K trade off against each other as you change K. Increasing K raises recall (you catch more relevant chunks) but lowers precision (you also include more irrelevant ones). The right operating point depends on your application: a system where missing one key fact causes a wrong answer should optimize for recall; a system where irrelevant context causes the model to hallucinate should optimize for precision.
NDCG@K
Normalized Discounted Cumulative Gain measures whether relevant chunks appear early in the ranked list. Chunks retrieved at position 1 receive full weight; chunks at position K receive logarithmically less. The result is normalized against the ideal ranking.
DCG@K = Σ (relevance_i / log2(i + 1)) for i in 1..K
IDCG@K = DCG@K of the perfect ranking
NDCG@K = DCG@K / IDCG@KExample: you have 2 relevant chunks for a query, with relevance scores [1, 1] (binary relevant/not). If they appear at positions 1 and 2, DCG = 1/log2(2) + 1/log2(3) ≈ 1.0 + 0.63 = 1.63. If they appear at positions 4 and 5, DCG = 1/log2(5) + 1/log2(6) ≈ 0.43 + 0.39 = 0.82. NDCG captures this difference; Precision@5 doesn't (it's 0.40 in both cases).
NDCG matters when your LLM attends more strongly to early context — which all transformer models with positional encodings do to some degree. A retrieval system that buries the right answer at position 8 out of 10 is worse than one that puts it first, even if Precision@10 is identical.
MRR (Mean Reciprocal Rank)
For each query, find the rank of the first relevant document. Take the reciprocal. Average across queries.
MRR = (1/|Q|) × Σ (1 / rank_of_first_relevant) for each query in QExample: across 3 queries, the first relevant document appears at ranks 1, 3, and 5. MRR = (1/3) × (1/1 + 1/3 + 1/5) = (1/3) × 1.533 ≈ 0.51.
MRR is the right metric when you only care about whether the single most relevant chunk makes it into the top results — common in question-answering pipelines where the LLM extracts a specific fact rather than synthesizing across multiple sources. It's insensitive to what happens after position 1, which makes it too coarse for multi-hop reasoning tasks.
Query: "What is the free-tier upload limit?"
Retrieved K=5: [B, A, C, D, E]
Relevant chunks: {A, C} (A at pos 2, C at pos 3)
Precision@5 = 2/5 = 0.40 (2 relevant out of 5 retrieved)
Recall@5 = 2/2 = 1.00 (both relevant chunks found)
NDCG@5 = 0.72 (relevant chunks not at top positions)
MRR = 0.50 (first relevant at position 2)
Same query, reranked: [A, C, B, D, E]
Precision@5 = 0.40 (unchanged — same chunks)
Recall@5 = 1.00 (unchanged)
NDCG@5 = 0.93 (relevant chunks now at top)
MRR = 1.00 (first relevant now at position 1)
The context stuffing failure mode
Most RAG implementations set K=10 or K=15 because "more context is better." This reasoning is wrong in two distinct ways.
First, irrelevant chunks in the context window actively hurt generation quality. Multiple controlled evaluations have found that injecting irrelevant documents into an LLM's context degrades its performance on questions about the relevant documents — sometimes below baseline (no retrieval at all). The model allocates attention to the irrelevant text and produces answers that blend the wrong content with the right content.
Second, the "lost in the middle" phenomenon: LLMs are systematically worse at using information that appears in the middle of a long context window compared to information at the beginning or end. If your retrieval returns 10 chunks and ranks them by cosine similarity, the most "similar" chunks (which are not necessarily the most relevant) appear first, and the truly relevant chunks may land in the middle of the context — exactly where the model will underutilize them.
The correct approach: optimize your K against measured Precision@K and Recall@K on your labeled eval set. For most enterprise RAG systems over structured documents, K=3 to K=5 with a reranking step outperforms K=10 with raw similarity scoring. Less context, better context.
Practical fix: hybrid retrieval plus reranking
The retrieval architecture that consistently outperforms pure dense retrieval in benchmarks:
Stage 1: hybrid retrieval (dense + sparse)
Run two retrieval systems in parallel and merge their result sets. Dense retrieval (your embedding model + vector store) excels at semantic matching — it finds relevant chunks even when the exact keywords aren't present. Sparse retrieval (BM25 or similar term-frequency methods) excels at exact-match queries — product names, error codes, version numbers, proper nouns that embeddings often handle poorly.
# Reciprocal Rank Fusion — merge dense and sparse rankings
def rrf(rankings: list[list[str]], k: int = 60) -> list[str]:
scores: dict[str, float] = {}
for ranking in rankings:
for rank, doc_id in enumerate(ranking):
scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank + 1)
return sorted(scores, key=scores.get, reverse=True)Reciprocal Rank Fusion combines the two ranked lists without requiring you to tune a weighting coefficient. Each document's RRF score is the sum of its reciprocal ranks across retrieval systems. The constant k=60 is the standard default — documents ranked very high in either system dominate the merged list.
Stage 2: cross-encoder reranking
Take the top-N candidates from Stage 1 (typically N=20–50) and score each (query, chunk) pair with a cross-encoder model. Unlike bi-encoders (which embed query and document independently), cross-encoders encode the query and document together — they can attend to the relationship between them. This is much more expensive per pair, which is why you only apply it to the candidate set, not the entire corpus.
from sentence_transformers import CrossEncoder
reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
pairs = [(query, chunk.text) for chunk in candidates]
scores = reranker.predict(pairs)
reranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)
final_chunks = [chunk for chunk, _ in reranked[:K]]The cross-encoder understands that "grayed out export button" and "export controls disabled" refer to the same problem. The bi-encoder didn't. This is the fix for the false-negative failure mode described earlier.
The reliability diagram equivalent for RAG: context precision vs. recall curves
For calibration, we plot predicted score against actual outcome rate. For RAG retrieval, the equivalent diagnostic is a Precision@K vs. Recall@K curve swept over K.
Precision
1.0 ┤
│ ● Cross-encoder rerank
0.8 ┤ ●
│ ●
0.6 ┤ ● Hybrid (dense+BM25)
│ ● ●
0.4 ┤ ● ●
│ ● ● Dense only (baseline)
0.2 ┤ ● ●
│ ● ●
0.0 ┼────────────────────────────
0.2 0.4 0.6 0.8 1.0 Recall
K=1,3,5,10,15,20 swept left to right on each curve
A system that dominates in this curve space — higher precision at every recall level — is unambiguously better. Most RAG teams have never generated this plot. They've looked at cosine similarity distributions and end-to-end generation accuracy, with nothing in between. That gap is where retrieval failures hide.
Generate this curve by sweeping K from 1 to 20, computing Precision@K and Recall@K against your labeled eval set at each K, and plotting the resulting (recall, precision) pairs. Do this for each retrieval variant. The curve that dominates across the operating range you care about is the system to deploy.
Building the eval dataset
The labeled eval set is the hard prerequisite for all of the above. In practice:
- Synthetic generation: for each document chunk, generate 2–3 plausible questions that this chunk (and only this chunk) answers. Use a capable LLM, then filter with a secondary model that verifies the chunk actually answers the generated question. Keep only pairs where the chunk contains a complete answer to the question.
- User query logs: if you have a deployed system, log real queries alongside the chunks that produced correct answers (verified via human review or LLM-as-judge). These are your hardest and most valuable eval examples because they represent real failure modes, not synthetic ones.
- Adversarial construction: deliberately add near-miss chunks — documents highly similar to query topics but answering different questions. These stress-test your system's false positive rate.
A dataset of 200–500 queries with labeled relevant chunks is enough to get reliable Precision@K and Recall@K estimates. You don't need thousands. You need coverage across your query distribution.
The evaluation harness for all of this — Precision@K, Recall@K, NDCG@K, MRR computation, hybrid retrieval, and the precision-recall curve generation — is in rag-eval.
↗ shadowmodder/rag-evalThe key shift: stop treating retrieval as a solved problem that happens before "the real work" of generation. In most RAG failures, retrieval is the problem. Measuring it correctly — with precision, recall, NDCG, and MRR against a labeled eval set — is the only way to know whether your pipeline is actually working.