Chunking: the decision that quietly caps your quality
How you split documents into passages sets a ceiling on how good your retrieval can ever be, and it is usually decided in five minutes by a default. This lesson covers the tension every strategy is trading against, and how to choose deliberately.
TL;DR: Chunking trades two things against each other: a chunk small enough to be precisely about one thing, and large enough to make sense without the text around it. Get it wrong and no model, reranker, or clever query rewriting recovers what you destroyed. Split on the document's own structure, not on a character count, and put the context back with metadata.
Where you are. You know the shape of a retrieval system. Splitting is the first of its five verbs and has the largest effect on the result. It is also the one most often set once, from a tutorial default, and never revisited. This lesson exists to stop you doing that.
Why this decides more than the model
A chunk is the unit of retrieval. Whatever you split your documents into is what gets embedded, stored, found, and handed to the model.
That means the chunk is the smallest thing your system can ever retrieve. If the answer to a question is spread across two chunks, you can only find half of it. If a chunk contains six unrelated topics, its position in meaning space is an average of six things and precisely about none of them.
This is why chunking sets a ceiling. Downstream improvements, better ranking, a stronger model, cleverer prompts, all operate on the chunks you produced. None of them can reassemble information you split apart or disentangle topics you merged.
The tension
Every chunking decision trades these two against each other.
Chunks want to be small. A short passage about one thing has a clean position in meaning space. Retrieval finds it precisely. It costs few tokens, so you can afford several.
Chunks want to be large. A passage needs enough around it to be understandable alone. "This does not apply to enterprise customers" is useless without knowing what "this" is. Split too finely and every chunk is a fragment referring to context that is no longer there.
There is no universal answer, because the right size depends on how your documents are written. That is the actual insight: the document's own structure usually tells you where to split, and it is a better guide than any number.
The strategies, and when each is right
Fixed size. Every N characters, often with an overlap so a sentence cut in half appears in both. Trivial to implement, and it is what most tutorials show.
It is the right choice for genuinely unstructured text with no headings or sections: transcripts, scraped prose, long chat logs. It is the wrong choice for anything with structure, because it will cut through the middle of a table, a code block, or a clause with no idea it has done so.
Structural. Split on what the document already uses: headings, sections, articles, clauses, slides, function definitions. Sizes come out uneven, which is fine and is usually a feature, because the author already decided what a self-contained unit was.
This is the right default for handbooks, documentation, contracts, and code. If your documents have headings and you are splitting on character counts, that is the first thing to change.
Semantic. Walk through the text and start a new chunk where the topic shifts, detected by comparing adjacent passages. More expensive to build and to run, and it earns its cost on long unstructured documents where structural splitting is unavailable and fixed size is destroying meaning.
Whole document. For short documents, do not split at all. A one-page policy is one chunk. Splitting it gains nothing and costs coherence.
Two things that fix most of the damage
Whatever you choose, two additions recover much of what splitting costs, and both are cheap.
Overlap. Repeat a little of the previous chunk at the start of the next. It cushions the boundary so an answer that straddles a split still appears in full somewhere. Modest overlap is standard; large overlap wastes storage and returns near-duplicate results, which brings its own problem.
Metadata and a restored heading. Store each chunk with its source, its section heading, its date, and its position. Two payoffs. You can filter before searching, which is often a bigger accuracy win than any ranking change. And you can prefix the chunk with its document title and heading, which turns "This does not apply to enterprise customers" back into something interpretable on its own.
That prefix trick is the highest-value cheap thing in this lesson. It costs a handful of tokens per chunk and it fixes the single most common cause of a retrieved passage being useless.
Do this before moving on
Take a document with real structure, a handbook page or a policy with headings.
Split it two ways: on its headings, and into fixed blocks of roughly 800 characters. Print the first six chunks from each.
Read the fixed-size ones and count how many begin mid-sentence, end mid-list, or contain a pronoun whose referent is in the previous chunk. Then ask a question that the document answers and decide, by eye, which set contains a chunk you would want handed to a model. This takes fifteen minutes and it usually settles the question permanently.
Go deeper
- Chunking is the reference: the strategies in more detail, plus size guidance and the trade-offs stated numerically.
- Practice question: How does contextual retrieval improve on plain chunking? is the heading-prefix idea taken further, where each chunk gets a short generated description of its place in the document.
- Hierarchical retrieval is the structural answer to the small-versus-large tension: retrieve on small chunks, return their larger parents.
- Practice question: Which chunking strategy would you choose, and why? is this lesson as an interview question, and a strong answer names the document type rather than a character count.
- Practice question: Your retrieval recall is low. How do you debug it? is where bad chunking shows up in production, usually after everyone has blamed the model first.
Key takeaways
- The chunk is the smallest unit your system can retrieve, so chunking sets a ceiling nothing downstream can raise.
- Small chunks retrieve precisely and lose their referents; large ones are self-contained and blur what they are about.
- Split on the document's own structure where it has any. Fixed size is for genuinely unstructured text.
- Overlap plus stored metadata, especially prefixing each chunk with its document title and heading, recovers most of what splitting costs.
Check yourself before an interviewer does. Answer from memory first.
A retrieved passage reads 'This does not apply to enterprise customers' and the model answers uselessly from it. What is the cheapest fix?