When retrieval is the wrong answer
Retrieval has become the reflex, and it is wrong often enough to be worth a lesson. Four situations where something simpler or entirely different is the better system, and the decision procedure that gets you there in ten minutes.
TL;DR: Retrieval is right when you have many documents, questions that hit unpredictable parts of them, and content that changes. It is wrong when everything fits in the prompt, when the real question is a database query, when what you need is a behaviour rather than a fact, and when a search box would have done. Reaching for it by default is the most common over-engineering in this field.
Where you are. You can build a retrieval system and diagnose it. Here is the counterweight, following the same principle as the agent course: know when not to build the thing. It will save you more time than any other lesson in this module.
Four times it is the wrong tool
1. Everything fits, comfortably and cheaply. You have a forty-page handbook, questions arrive a few dozen times a day, and the whole thing costs little to include in the prompt.
Building retrieval here buys you a pipeline to maintain, an index to keep current, and a new failure mode where the right passage is not found. Putting the document in the prompt has none of those, and it cannot miss.
The line moves with your volume and document size, so do the arithmetic from module 2 rather than assuming. Occasional questions over a small corpus is genuinely the case where retrieval is over-engineering.
2. The question is really a database query. "How many enterprise customers churned last quarter", "which contracts expire in March", "what is the average resolution time".
These are aggregations over structured data. Retrieval finds passages, and no amount of chunk tuning makes it count. If the answer lives in a database, query the database. The interesting AI problem here is translating the question into that query, which is a different architecture and a genuinely good use of a model.
The tell: the answer requires counting, comparing, or ranking across many records rather than reading one passage.
3. You need a behaviour, not a fact. You want a particular tone, a house format, or adherence to a convention that is hard to state as rules.
Retrieval supplies facts. If the gap is how the model does something rather than what it knows, examples in the prompt are the cheap fix and adapting the model is the expensive one. Neither is retrieval, and teams do occasionally build a retrieval system to fix an output-style problem.
4. A search box would have done. Sometimes the user wants the document, not a paragraph about it. Lawyers, engineers, and anyone with a compliance obligation frequently prefer to be handed the source and read it themselves.
Adding a generation step on top can subtract value: it introduces a paraphrase they now have to verify, for a job that plain search already did. Worth asking whether the user wants an answer or a document.
The decision, in order
Ten minutes with this diagram before building is worth several weeks.
When it genuinely is right
To be clear, because the previous four sections argue one way: retrieval is the right answer often, and it is the right answer whenever all of these hold.
- There is more material than fits, or would be affordable to include, at your volume.
- Questions hit unpredictable parts of it, so you cannot pre-select what to include.
- The content changes, and you need answers to change with it without retraining anything.
- Answers must be attributable to a source someone can check.
That combination is common, which is why this pattern won. It is just not universal, and the four cases above are frequent enough that the reflex costs real time.
The honest summary of this module
Retrieval is the most useful architecture you will learn in this course, and most of the work in it is not AI work. Splitting documents sensibly, indexing them with dates, running two kinds of search, ranking the results, keeping the corpus current, and deciding when to decline. The model reads the page you found.
Do that part well and an ordinary model gives excellent answers. Do it badly and the best model available gives confident answers from the wrong page.
Do this before moving on
Take the project you named back in module 1 and walk it through the diagram above, out loud or on paper, answering each question with a reason rather than an instinct.
If you end at retrieval, you now have a justified architecture and you are ready to build. If you end anywhere else, you have just saved yourself weeks, and the next lesson is the build guide for whenever you do land on retrieval.
Go deeper
- Retrieval versus long context is case 1 argued with the cost curves, and the most useful page here if your corpus is small.
- RAG over SQL and text-to-SQL is case 2 done properly, and it is a common interview question because so many teams get it wrong.
- Fine-tuning versus RAG versus prompting is the decision this whole lesson is a practical version of, including where case 3 lands.
- Practice question: When would you use a long context window instead of RAG? is case 1 as a system-design probe, and "it depends" scores nothing without the arithmetic.
- Practice question: How do you improve retrieval quality on a domain corpus? is where you go once you have decided retrieval is right and want it to be good.
Key takeaways
- Retrieval earns its complexity when there is too much material to include, questions are unpredictable, content changes, and answers must be attributable.
- If the question requires counting or comparison across records, it is a database query, and no chunk tuning fixes that.
- If the gap is behaviour rather than knowledge, the answer is examples or adaptation, not retrieval.
- Most of retrieval quality is data and pipeline work. The model reads the page you found.
Check yourself before an interviewer does. Answer from memory first.
A team wants retrieval over a 30-page policy handbook receiving about fifty questions a day. What should you suggest first?