RAG Evaluation Strategies That Actually Matter
Most teams building RAG systems spend the bulk of their time on retrieval and generation, then ship to production and wonder why users aren't happy. The evaluation layer is where that gap closes. And it's harder than it looks.
This is a practical walkthrough of how to evaluate a RAG pipeline end to end, from retrieval quality all the way to generation faithfulness. No hand-waving, no toy examples.
Why RAG Evaluation Is Its Own Problem
Standard NLP benchmarks don't cut it here. RAG systems fail in two distinct places: the retriever pulls the wrong context, or the generator hallucinates on top of correct context. These are different failure modes with different fixes, so you need evaluation strategies that surface each one separately before you can treat them together.
The other complication is that RAG systems are inherently stateful. The quality of an answer depends on what documents exist in the store, how they were chunked, how embeddings were generated, and what the query looked like after rewriting. A single accuracy number doesn't tell you where in that chain things broke.
Part 1: Retrieval Evaluation
Before you even think about the LLM, you need to know whether your retriever is pulling the right documents.
Recall@K
Recall@K asks a simple question: of all the relevant documents for this query, how many appear in the top K results? This is usually your first signal. If Recall@5 is low, your retriever is losing chunks that matter, and no amount of prompt engineering will fix a downstream answer built on missing context.
You need a labeled dataset to compute this, which we'll cover in Part 3.
MRR (Mean Reciprocal Rank)
Recall@K tells you that a relevant document appeared somewhere in the top K results. MRR tells you where. A relevant document ranked first is far more useful than one ranked fifth, because most LLMs attend more strongly to context that appears early in the prompt. MRR rewards getting the best document to the top.
NDCG (Normalized Discounted Cumulative Gain)
NDCG is the most nuanced retrieval metric. It accounts for graded relevance rather than binary yes/no, and it penalizes relevant documents appearing lower in the ranking more than ones near the top. If your use case involves documents with varying degrees of relevance (highly relevant vs. marginally relevant), NDCG gives you a more honest picture than Recall@K alone.
In practice: start with Recall@K to catch obvious retrieval failures, then layer in MRR and NDCG once you have a stable baseline.
Part 2: Generation Evaluation
Once you've confirmed the retriever is returning useful context, you need to evaluate what the LLM does with it.
Faithfulness
Faithfulness measures whether every claim in the generated answer is actually supported by the retrieved context. A faithful answer doesn't introduce facts the retriever never provided. This is the metric most directly tied to hallucination.
Frameworks like RAGAS compute faithfulness by decomposing the answer into atomic claims and checking each against the context. It's not perfect, but it's a practical proxy for "did the model make something up."
Answer Relevance
This one asks whether the answer actually addresses the user's question, regardless of whether it's faithful to the context. You can have a perfectly faithful answer that's still unhelpful because it answers the wrong thing. Answer relevance catches that.
A low answer relevance score often points back to query understanding failures rather than retrieval or generation issues.
Context Precision and Context Recall
These two metrics evaluate the relationship between the retrieved context and the final answer from the other direction. Context precision asks how much of the retrieved context was actually used. Context recall asks whether the generated answer covered everything in the relevant context. Both matter when you're trying to understand whether your chunking strategy is creating noise or leaving useful information on the table.
Part 3: Building a Golden Dataset
All of the above requires ground truth. Without it, you're flying blind.
A golden dataset is a curated set of (query, relevant document, expected answer) triples. Building one is the least glamorous part of RAG evaluation, but it's foundational.
A few approaches that work:
Synthetic generation. Use an LLM to generate questions from your document corpus, then manually review and filter. This gets you coverage fast, but you need a human pass to catch questions the model generated poorly or questions that are unanswerable from the documents.
Production query sampling. Take real queries your users are already sending, identify which ones are representative, then annotate the relevant documents and reference answers. These are the highest-signal examples because they reflect actual user intent.
Adversarial test sets. Add queries designed to trip the system up. These include ambiguous queries, queries with no good answer in the corpus, queries where multiple contradictory documents exist, and queries that look similar but require different answers. Adversarial examples surface brittleness that a clean golden dataset won't catch.
Part 4: LLM-as-Judge
Manually annotating hundreds of answers at evaluation time doesn't scale. LLM-as-judge is the practical solution: use a separate LLM (often a stronger model than the one you're evaluating) to score answers on faithfulness, relevance, and correctness.
This works surprisingly well when you invest in the scoring rubric. A vague "rate this answer 1-5" produces noisy scores. A structured rubric with explicit criteria for each score level produces scores that correlate well with human judgment.
Two mature frameworks here:
RAGAS provides out-of-the-box metrics for faithfulness, answer relevance, context precision, and context recall. It's designed specifically for RAG and works well as a starting point.
TruLens takes a more flexible approach, letting you define custom feedback functions and track them across your full evaluation pipeline. It's more work to set up but gives you more control.
For teams already in the LangSmith ecosystem, LangSmith custom evaluators let you bring LLM-as-judge scoring directly into your existing observability workflow, which keeps your evaluation data alongside your traces rather than in a separate system.
Part 5: Integrating Evaluation Into CI/CD
Ad hoc evaluation runs are better than nothing, but they won't catch regressions. The goal is making evaluation a gate in your deployment pipeline.
The pattern that works: maintain a versioned golden dataset, run your full evaluation suite on every PR that touches the retrieval or generation layers, and block merges that drop key metrics below a defined threshold. Recall@K and faithfulness are usually the best candidates for hard gates. Softer metrics like NDCG can be tracked for trend rather than used as blockers.
LangSmith makes this straightforward if you're already using it for tracing. You can push evaluation results as experiments, compare across runs, and surface regressions in the same place you're already monitoring production behavior.
Putting It Together
RAG evaluation isn't one thing. It's a stack of concerns across retrieval, generation, and context quality, each of which requires different metrics and different tooling.
The minimum viable setup:
- A golden dataset with at least 50 to 100 examples covering your real query distribution
- Recall@K and MRR to track retrieval quality
- Faithfulness and answer relevance via RAGAS or TruLens for generation quality
- A CI step that runs the evaluation suite and surfaces regressions before they hit production
Start there. As your system matures, layer in adversarial examples, context precision/recall, and custom LLM-as-judge rubrics tailored to your domain. The teams that do this consistently are the ones whose RAG systems actually improve over time instead of drifting.
Questions or pushback? Always happy to dig into specific setups.