Why your search misses the obvious answer
Pure vector search fails in two specific, predictable ways, and both have standard fixes that most first systems are missing. This lesson covers the two-stage design that turns a demo into something that finds the right passage.
TL;DR: Vector search fails on exact terms, because an error code is not semantically similar to anything, and it fails on relevance, because it ranks by aboutness. The fixes are complementary and both cheap: run keyword search alongside vector search and merge, then rerank the merged shortlist with something slower and more accurate. Retrieve broad, rank narrow.
Where you are. You know what retrieval is for and how the units get made. The search step is where most first systems have their biggest and most fixable gap. If you build one thing from this module, build the two-stage design below.
The two failures
Your first system embeds the question, finds the nearest chunks, and hands them over. It works, until it does not, and the two ways it does not are specific enough to name.
Failure 1: exact terms. A user searches for error code E-4471, an invoice number, a product SKU, or a surname. None of these mean anything. There is no semantic neighbourhood for an identifier, so the embedding lands somewhere arbitrary and the passage that literally contains the string does not come back. Old-fashioned keyword search would have found it instantly.
Failure 2: relevance. You met this in the embeddings lesson: similarity is aboutness, not relevance. Search "can I expense a taxi" and every passage about expenses scores highly, including the outdated policy, the one about mileage, and the one saying taxis are not covered. All genuinely about expenses. One answers the question.
The two failures need different fixes, which is why systems that fix only one still disappoint.
Fix 1: search both ways and merge
Keyword search is not obsolete. It is excellent at exactly what vector search is worst at, and vice versa.
| Keyword search | Vector search | |
|---|---|---|
| Exact identifiers, codes, names | strong | weak |
| Rare technical terms | strong | weak |
| Paraphrase, synonyms, different wording | weak | strong |
| Questions phrased unlike the document | weak | strong |
Run both, merge the results, and you cover both columns. The merging deserves one note, because it is where people improvise badly: the two scores are not comparable. A keyword score and a distance are different units, so averaging them is meaningless.
The standard solution ignores the scores and uses only the ranks. Something that appears high in either list ranks high in the merged one, and something appearing in both ranks higher still. It is a few lines of code, needs no tuning, and holds up better than any weighting you would pick by hand.
Fix 2: rerank the shortlist
The second failure needs something that judges relevance rather than similarity, and the reason vector search cannot is worth understanding, because it explains the whole design.
For search to be fast, every chunk must be embedded before any question arrives. So each chunk is positioned without knowledge of the question, and at question time you compare two positions computed independently. That is fast, and it is why a million chunks are searchable in milliseconds. It is also why it can only measure aboutness: nothing ever looked at the question and the passage together.
A reranker does exactly that. It takes the question and one passage as a pair and scores how well that passage answers that question. Much more accurate, and far too slow to run over a million chunks.
So you use both, in stages:
Retrieve broad, rank narrow. Stage one is cheap and errs toward including things, casting a wide net over everything. Stage two is expensive and accurate, and only ever sees a shortlist.
This is the single highest-return change available to a working retrieval system. Teams routinely find reranking moves quality more than switching embedding models, changing chunk size, or upgrading the generation model, and it is a small amount of work.
The number that is usually wrong
One more decision: how many chunks to put in the prompt.
The instinct is that more is safer. Module 2 already told you why it is not. Attention is shared, so each extra passage dilutes the share going to the one that mattered, and every passage costs tokens on a curve that rises faster than linearly.
With reranking in place you can afford to send fewer, because the top few are genuinely the best few rather than merely the nearest. That is the second benefit of a reranker and the one people notice on the bill: better answers from a shorter prompt.
Start around five and tune it against your example set from module 3, in both directions. Many systems are sending twenty when eight would answer better and cost less.
Do this before moving on
Take the retrieval you have, or a simple vector search over your own documents, and run three queries against it: one containing an identifier or code, one phrased in words the document does not use, and one where several passages are about the subject but only one answers.
Note which of the three fails and how. Typically the first fails outright and the third returns a plausible wrong passage at rank one. Those are the two failures in this lesson, on your own data, in ten minutes.
Go deeper
- Hybrid search is fix 1 in detail, including why rank-based merging beats score weighting.
- Reranking is fix 2: how the two model types differ, what a rerank costs in latency, and how to size the shortlist.
- Vector search covers how the fast stage finds nearest neighbours without comparing everything, which is worth knowing before you scale.
- Practice question: How would you combine keyword and vector retrieval? is fix 1 as an interview question, and naming rank-based fusion is what a strong answer sounds like.
- Practice question: Everything scores as highly similar but nothing is relevant. What now? is failure 2 exactly, and it is a favourite because it separates people who have run a system from people who have read about one.
Key takeaways
- Vector search misses exact identifiers, and it ranks by aboutness rather than by whether a passage answers the question.
- Run keyword and vector search together and merge on rank, not on score. The two scores are different units.
- A reranker scores the question and passage as a pair, which is why it judges relevance and why it is too slow to run over everything.
- Retrieve broad, rank narrow, then send fewer passages. Reranking usually beats changing models or chunk sizes.
Check yourself before an interviewer does. Answer from memory first.
Users searching for error code E-4471 get nothing useful, though the passage containing it exists. Why?