TL;DR: Evaluate the two stages separately. For retrieval, build a labeled set and measure recall@k, precision@k, and nDCG (did the right chunks come back). For generation, measure faithfulness/groundedness (is every claim supported by the retrieved context), answer relevance, and correctness, using an LLM judge plus human spot-checks. Decomposing this way tells you whether a failure is a retrieval problem or a generation problem, which determines the fix.
How to approach it. Insist on splitting retrieval from generation, because a bad answer can come from either and the fix differs. Acknowledge there is rarely a single gold answer, so you lean on reference-light metrics (faithfulness against the retrieved context) plus a small labeled set, and you calibrate any LLM judge against humans.
A strong answer. The system has two stages and each gets its own scorecard. Mixing them is how teams spend a week tuning a reranker when the real bug was a verbose generator.
| Stage | Metric | What it tells you | Needs labels? |
|---|---|---|---|
| Retrieval | recall@k | Did the relevant chunks make the top k? This is the ceiling on answer quality. | Yes (query to relevant-chunk) |
| Retrieval | precision@k, nDCG | How clean and well-ranked the top k is, which is what the generator consumes. | Yes |
| Generation | faithfulness / groundedness | Is every claim supported by the retrieved context? Direct hallucination signal. | No (answer + sources only) |
| Generation | answer relevance | Does it address the question or just echo context? | No |
| Generation | correctness | Right vs wrong against a curated reference. | Yes (reference answers) |
The standout is faithfulness: it needs no gold answer, only the answer and its sources, so an LLM judge can check each claim against the context and flag the unsupported ones. recall@k is the other anchor, because if the right chunk never gets retrieved, no generator can recover.
Method. Build a representative eval set from real production queries, with labeled relevant chunks and, where feasible, reference answers. Use an LLM judge for faithfulness and relevance at scale, but calibrate it against human labels on a sample (judges carry position, verbosity, and self-preference biases). Gate every chunking, embedding, reranker, or prompt change on these numbers, and track them over time so drift surfaces before users find it.
Decomposition pays off. A wrong answer with good retrieval is a generation or prompt problem; a wrong answer with poor recall is a retrieval problem (chunking, embeddings, hybrid search). Without the split you cannot localize the failure, and you tune blindly.
Key takeaways
- recall@k is the hard ceiling: retrieval misses cannot be fixed downstream, so measure it first.
- Faithfulness is the one reference-free metric that directly catches hallucination, scored per-claim against the retrieved context.
- An LLM judge is fast but biased; calibrate against humans and randomize answer order before trusting its scores.
- Offline eval gates releases, online signals (thumbs, task success, deflection) confirm real impact.
What interviewers probe next.
- "How reliable is LLM-as-judge?" Useful at scale, biased; calibrate against humans, randomize answer order, and audit a sample. Do not trust raw judge scores as truth.
- "No labeled data yet, how do you start?" Generate synthetic Q&A from the corpus to bootstrap retrieval labels, then curate with humans; mine production queries as they arrive.
- "Online vs offline eval?" Offline gates releases; online (thumbs up/down, task success, deflection rate) measures real impact, since offline wins do not always hold.
- "How do you catch a regression from a 'better' embedding model?" recall@k on the fixed eval set; higher benchmark scores elsewhere do not guarantee your corpus improves.
Common mistakes.
- Shipping with no eval set and judging by anecdote.
- Measuring only end-to-end answer quality, so you cannot localize whether retrieval or generation failed.
- Trusting an uncalibrated LLM judge as ground truth.
- Optimizing recall@k while ignoring faithfulness, so the model is fed good context and still hallucinates.