How a working retrieval system goes wrong
The failures that arrive after launch, once real documents and real questions replace your test set. Six of them, each with the signal that identifies it, because the wrong diagnosis here wastes weeks.
TL;DR: Six failures account for most production disappointment: the answer was never retrieved, it was retrieved and ignored, the source was stale, two sources disagreed, the answer needed several documents combined, and the system answered when it should have declined. The first diagnostic question is always the same: was the right passage in what we handed over?
Where you are. You can build a retrieval system that works on the documents you tested with. What happens next is written here to be returned to. Read it once now so you recognise these when they arrive, because the expensive mistake is diagnosing the wrong one and spending a week rebuilding the stage that was fine.
Split the failure first
Before anything else, answer one question: was the correct passage in the set handed to the model?
Everything downstream depends on it, it takes one log line to answer, and teams routinely spend weeks tuning prompts for what turns out to be a retrieval miss, or swapping embedding models for what turns out to be a generation problem.
Log the retrieved chunk ids with every request from the first day. Without them this question is unanswerable and you are guessing.
The six
1. It was never retrieved. The answer exists in your documents and did not come back.
Signal: search the corpus by hand, find the passage, confirm it was absent from the retrieved set. Causes, in order of likelihood: the answer was split across two chunks; the question uses different words and there is no keyword search; the document was never indexed; a metadata filter excluded it.
2. It was retrieved and ignored. The right passage was handed over and the answer contradicts it or omits it.
Signal: the answer is wrong, and the correct passage is sitting in your logs. Causes: it was ranked fifth of twenty and buried in the middle of a long prompt; another retrieved passage was more assertive and won; the prompt does not say the answer must come from the supplied material.
3. The source was stale. A perfectly correct answer, from last year's policy.
Signal: the answer traces to a real passage that is out of date. Causes: superseded documents were never removed, and nothing prefers recency. This is the failure most likely to reach a customer, because it survives every quality check that only asks whether the answer was grounded. Fix: index dates, filter or prefer by them, and delete superseded documents rather than keeping them for reference.
4. Two sources disagreed. Your documents contain contradictions and the model picked one, silently.
Signal: the answer is defensible and there is an equally defensible opposite in the corpus. Fix: this is not primarily a technical problem. Retrieval surfaces contradictions your organisation already had, and the durable fix is having one authoritative document. Technically, you can detect conflicting retrieved passages and surface both rather than choosing, which is usually the honest behaviour.
5. The answer needed several documents. "How many customers on the enterprise plan asked about billing last quarter" cannot be answered from any single passage.
Signal: the question requires counting, comparing, or combining across documents. Fix: recognise that this is not a retrieval problem. Retrieval finds passages; it does not aggregate. This needs a query against structured data, which is module 5 territory when the model has to choose to run it. Trying to solve it with more chunks retrieved is a trap that wastes real time.
6. It answered when it should have declined. Nothing relevant was found, and an answer appeared anyway.
Signal: a confident answer with retrieved passages that do not support it. Fix: a relevance floor, so that if nothing scores above a threshold you return "I could not find that" rather than passing weak matches on. This is the same abstention lesson from module 2, and it is the single most trust-preserving behaviour a retrieval system has.
The pattern
Notice how few of these are fixed by a better model. One is a chunking problem, two are search and indexing problems, one is a data governance problem, one is an architecture mismatch, and one is a policy decision about when to answer at all.
That is the module's argument arriving: retrieval quality is mostly a data and pipeline problem wearing an AI hat. The model is the last stage and rarely the limiting one.
Do this before moving on
Take ten questions your system gets wrong, or ten you expect it to. For each, log what was retrieved and answer the split question: was the right passage there?
Sort your ten into the two piles. The proportions tell you what to work on next, and they are usually not what the team assumed. A system with eight retrieval misses and two generation problems should not be having a prompt-engineering discussion, and until you do this you will not know which one you have.
Go deeper
- Citations and grounding is the mechanism that makes failures 2 and 6 detectable rather than invisible.
- RAG freshness is failure 3 as an interview question, and it comes up more than people expect because it is the failure that reaches customers.
- Handling conflicting sources is failure 4, and the strong answer notices it is partly an organisational problem.
- Practice question: How do you evaluate a RAG system? is how you would find these six systematically rather than one complaint at a time. The direct bridge to module 6.
- Practice question: How do you verify a citation actually supports the claim? is the programmatic version of the split question above, and where this gets genuinely rigorous.
Key takeaways
- Ask first whether the right passage was retrieved. It splits every failure into two piles with different fixes, and it needs one log line.
- The six: never retrieved, retrieved and ignored, stale source, contradictory sources, needs aggregation, and answered when it should have declined.
- Stale sources are the failure most likely to reach a customer, because a stale answer is still a grounded one.
- Most of these are data and pipeline problems. The model is the last stage and rarely the limiting one.
Check yourself before an interviewer does. Answer from memory first.
What is the first thing to establish when a retrieval-backed answer is wrong?