Project: build a retrieval system over your own documents
The first project of the course. Build a working question-answering system over documents you actually have, in five stages, each one checkable on its own, and finish with something honest enough to put in a portfolio.
TL;DR: Five stages, roughly four hours, ending with a system that answers questions from your own documents, cites what it used, and says so when it does not know. Each stage is independently checkable, so stopping halfway still leaves you with something that runs.
What you will have built
A question-answering system over a corpus you care about, that retrieves relevant passages, answers only from them, shows which passages it used, and declines when nothing relevant is found.
That last clause is what makes it worth having. Plenty of people have built the first three-quarters of this. A system that knows when to say "I could not find that" is the one you can demonstrate without hedging, and it is the difference between a demo and something a colleague could use.
Before you start
Lessons: all of module 4, plus the tokens and embeddings lessons from module 2. The evidence lesson from module 3 matters at stage 5.
What you need: a language model API, an embedding model (often the same provider), and something to store vectors. For a first build the store can be an in-memory list and a loop computing distances. A real vector database is a stage-6 concern and adds nothing to what you learn here.
Documents: at least twenty, ideally a hundred, that you genuinely know. Your own notes, a handbook, project documentation, saved articles. Knowing the corpus matters more than its size, because you have to be able to tell whether an answer is right.
The stages
Stage 1: load and split
Read your documents and split them into passages. Use structure if they have any, headings or sections, and fixed size with overlap if they do not. Store each chunk with its source file, its heading, and its position.
Check: print twenty chunks at random and read them. Can each be understood alone? If a chunk starts mid-sentence or refers to something that is no longer there, fix the splitting before continuing. Everything downstream inherits this.
Stage 2: embed and store
Embed each chunk and keep the vectors with their metadata. Write a function that takes a question, embeds it, and returns the nearest chunks.
Check: query it with five questions you know the answers to and read what comes back, without involving a model yet. This is the stage people skip, and it is the one that tells you whether anything downstream can work. If the right passage is not in the top ten here, no prompt will save you.
Stage 3: answer from the passages
Send the question and the top passages to a model, with instructions to answer only from what is supplied and to name which passages it used. Return the answer together with its sources.
Check: for five questions, verify each cited passage genuinely supports the claim. Read them. A citation that does not quite say what the answer says is one of the commonest faults at this stage, and finding one yourself is more instructive than being told they exist.
Stage 4: decline when unsure
Add a relevance floor. If the best retrieved passage scores below a threshold, return "I could not find that in these documents" and skip the model call entirely.
Pick the threshold by running a handful of questions you know are unanswerable from your corpus, looking at the scores, and choosing something below the worst genuine match and above the best spurious one. It will be imperfect. Imperfect and present beats absent.
Check: ask three questions your documents plainly do not cover. All three should decline. If any produces an answer, the floor is too low, and this is the single most valuable check in the project.
Stage 5: measure it
Build the example set from module 3: twenty questions with the answers you would accept, weighted toward awkward cases. Score your system. Write the number down.
Then change one thing, add keyword search alongside vector search, or add a reranking pass, and score again.
Check: you can state what your system scores and what one specific change did to it. That sentence is what separates this from a weekend demo, and it is what an interviewer is listening for.
Honest scope
Roughly four hours if things go smoothly, and a first attempt rarely does. Stage 1 usually takes longer than expected because real documents are messier than expected.
Deliberately not included: a real vector database, incremental re-indexing when documents change, authentication, a user interface, or serving it to anyone else. Those are production concerns and module 7 material, and adding them now obscures the parts that teach you something.
Do not use a framework for this one. Writing the five steps yourself is the point. Frameworks are worth using afterwards, and you will understand what they are doing for you, which is exactly the position you want to be in when one of them behaves unexpectedly.
Extensions, if you want to go further
- Add keyword search and merge on rank, then measure whether it helped. It usually helps most on identifiers and names.
- Add a reranking pass over the top fifty and cut the passages sent from ten to five. Measure both quality and cost.
- Index dates and prefer recent sources, then test with a corpus containing a superseded document.
- Detect when two retrieved passages disagree and surface both rather than choosing.
Go deeper
- RAG pipeline is the reference architecture to check your build against once it works.
- Vector search is what to read before replacing your loop with a real index at scale.
- Reranking is the first extension above, and usually the highest-return one.
- Practice question: Build an indexer and a retriever is this project as an interview exercise, sometimes given as a take-home.
- Practice question: How do you evaluate a RAG system? is stage 5 done rigorously, and the bridge into module 6.
Key takeaways
- Check each stage on its own. Most retrieval systems fail at stage 2, and every later stage inherits it.
- Read your chunks and read your retrieved passages before involving a model. Both are ten-minute checks that save days.
- A system that declines when nothing relevant is found is worth more than one that is right slightly more often and confidently wrong the rest of the time.
- Finish with a number and one measured change. That is what makes it a project rather than a demo.
Check yourself before an interviewer does. Answer from memory first.
Your system gives poor answers. Which stage should you check before touching the prompt?