TL;DR: There is no universal chunk size; start around 400-600 tokens with ~10-15% overlap, respect document structure, and tune against retrieval recall on a labeled set. Use hybrid retrieval: dense embeddings for semantic and paraphrase matches, BM25 for exact terms, codes, names, and rare tokens that embeddings blur. Fuse the two and rerank with a cross-encoder.
How to approach it. Resist giving a single magic number. Explain what each knob trades off and that you tune chunking empirically against recall@k, not by feel. Then contrast what dense and sparse retrieval each capture, which is the real test.
A strong answer. Chunking. Too small and a chunk loses the context needed to answer (and you fragment a single fact across chunks); too large and the embedding averages over many topics, diluting relevance, plus you burn context-window budget feeding the generator. A sane default is 400-600 tokens with ~10-15% overlap so a fact straddling a boundary survives, and chunking that respects structure (headings, paragraphs, code blocks) rather than blind fixed windows. The honest answer is you tune it: build a labeled query-to-relevant-chunk set and sweep chunk size and overlap against recall@k. Domain matters: legal contracts chunk differently than chat logs.
Dense vs sparse. Dense retrieval embeds query and document into a vector space and matches by cosine similarity; it captures semantic similarity and paraphrase ("how do I reset my password" matches "credential recovery steps"). Its weakness is exact, rare, or out-of-vocabulary tokens: error codes, SKUs, person names, and API method names get blurred into nearby concepts. BM25 (sparse, lexical) is the mirror image: it nails exact-term and rare-token matches but misses synonyms and paraphrase.
| Dense (embeddings) | Sparse (BM25) | |
|---|---|---|
| Wins on | Paraphrase, synonyms, semantic intent | Exact tokens, codes, SKUs, names, rare terms |
| Fails on | Rare strings, IDs, OOV tokens | Synonyms, reworded queries |
| Match basis | Cosine in vector space | Lexical term overlap (TF-IDF style) |
| Index | ANN (HNSW) | Inverted index |
Hybrid retrieval runs both and fuses the results (Reciprocal Rank Fusion is a simple, strong default that needs no score calibration), then a cross-encoder reranks the union for precision. That combination is the workhorse because real queries mix conceptual and exact-match intent in the same sentence.
The defensible position: default to hybrid plus a reranker. Pure dense looks fine in demos and then fails the moment a user pastes an error code or a product name.
Key takeaways
- No universal chunk size: default to 400-600 tokens with 10-15% overlap, respect structure, then tune against recall@k.
- Dense captures meaning and paraphrase; BM25 captures exact and rare tokens; each fails where the other wins.
- Hybrid plus RRF plus a cross-encoder reranker is the default; pure dense quietly drops codes, IDs, and names.
- Validate chunking and retrieval with a labeled set and recall@k, and re-tune when the corpus or query mix shifts.
What interviewers probe next.
- "Why not just a bigger embedding model for exact matches?" Embeddings fundamentally compress; rare tokens and exact strings are where lexical search structurally wins. Hybrid is cheaper than fighting that.
- "How do you fuse dense and sparse scores?" RRF (rank-based, no score calibration needed) or a weighted or learned combination; RRF is insensitive to score scales and parameter-light.
- "Overlap downside?" Duplicate content inflates the index and can return near-duplicate chunks; dedupe in reranking.
- "How do you know chunking is right?" Recall@k on a labeled set, not vibes; re-tune when the corpus or query mix changes.
Common mistakes.
- Quoting a fixed chunk size as gospel instead of tuning against recall.
- Pure dense retrieval, then losing every exact-match query (codes, names, IDs).
- Optimizing recall@k alone and feeding a weak reranker, so the generator still gets noise.
- Ignoring document structure and splitting mid-sentence or mid-table.