RAG Evaluation

TL;DR: Evaluate a RAG system in two parts. Retrieval: did the relevant chunks get fetched? Measure recall@k (the ceiling, if the chunk is not retrieved, no generation can recover), plus precision and rank metrics. Generation: given the retrieved context, is the answer faithful (each claim supported by the context, see citations) and correct/helpful? Separating the two tells you which half failed, retrieval (improve chunking/embeddings/reranking) or generation (improve grounding/prompting). Most RAG failures are retrieval failures.

Why evaluate the halves separately

A RAG answer can be wrong for two very different reasons: retrieval failed (the right context was never fetched, so the model never had a chance) or generation failed (the context was there but the model used it poorly). If you only score the final answer, you cannot tell which, and you will fix the wrong thing. Teams that skip this step spend a sprint tuning prompts when the real problem was a chunk size that split the answer across two passages neither of which crossed the retrieval threshold. So evaluate retrieval and generation as separate stages.

Retrieval metrics

• Recall@k: of the relevant chunks, how many appear in the top k. This is the ceiling for the whole system, the relevant chunk must be retrieved or generation cannot recover, so it is usually the most important retrieval metric.
• Precision@k: how much of the top-k is relevant (noise distracts the model and wastes context).
• MRR / nDCG: the rank of the relevant chunks (the model attends more to earlier context).

Build a small eval set of (query, relevant-chunk) pairs, human-labeled or LLM-generated from your documents. A few hundred labeled queries is enough to move from anecdote to signal.

Worked example: localizing the failure

Take a 300-query eval set on an internal docs bot. You log recall@5 and a faithfulness score per answer:

End-to-end correctness (0.58) sits just under recall@5 (0.62), and faithfulness is high (0.91). The diagnosis is unambiguous: this is a retrieval problem. Spending effort on prompt engineering would move 0.91 toward 0.95 at best, gaining maybe two points end-to-end. Fixing chunking and adding a reranker to push recall@5 from 0.62 to 0.85 is worth roughly 25 points. Measure first, then you know where the 25 points live.

Generation metrics

• Faithfulness / groundedness: is each claim supported by the retrieved context? Check with an entailment model or an LLM judge, and require citations so unsupported claims are visible. This is the key RAG-generation metric, an unfaithful answer is a hallucination even with good retrieval.
• Answer relevance / correctness: does it actually answer the question, judged against references or by an LLM judge.

Frameworks like RAGAS compute context recall/relevance plus faithfulness and answer relevance. TruLens and DeepEval cover similar ground. As always, offline evals gate; an online A/B test confirms real impact.

Why interviewers probe this

"My RAG gives mediocre answers, what do you do?" is answered by separating retrieval from generation and finding which failed, the core RAG debugging skill. A strong answer stresses recall@k as the ceiling (most failures are retrieval failures), names faithfulness as the key generation metric, and builds a labeled eval set, gating with offline evals and confirming with an online A/B. The reserved follow-up is usually "how do you build the eval set without hand-labeling thousands of queries?" The strong response: generate candidate (query, chunk) pairs from your own documents with an LLM, then human-verify a sample to estimate label noise. That is the difference between guessing and systematically improving a RAG system.

Common misconceptions

• "Just score the final answer." You cannot tell whether retrieval or generation failed; evaluate them separately.
• "Recall@k is a minor metric." It is the system ceiling, if the chunk is not retrieved, generation cannot recover.
• "A fluent answer is a good answer." Without faithfulness checks it may be unsupported; grounding and citations matter.
• "Offline RAG metrics are enough." They gate; an online A/B confirms real-world quality.

Key takeaways

• Evaluate retrieval and generation separately so you know which half failed.
• Retrieval: recall@k (the ceiling), precision, and rank metrics, on a labeled (query, chunk) set.
• Generation: faithfulness (claims supported by context) and answer quality, via citations and LLM judges.
• Most RAG failures are retrieval failures; offline evals gate, an online A/B confirms.