A big context window does not mean you should fill it
Context windows keep growing, and the instinct is to stop thinking about what goes in them. This lesson closes the module by showing why capacity is not recall, what actually degrades as the window fills, and how to decide between more context and better context.
TL;DR: The context window tells you what fits, not what the model will use well. Quality degrades as the window fills, cost rises faster than the token count, and the middle of a long input is the weakest position in it. Treat the window as a budget you spend deliberately, not a bucket you fill, and the rest of this course follows naturally.
Where you are. This closes the module. You know text becomes tokens, that meaning becomes position, that attention costs grow quadratically, that output is a plausible continuation, and that the last step is configurable. This lesson puts those together into the decision you will make more often than any other: what goes in the prompt.
Capacity is not recall
The window is a hard limit on how many tokens the model can attend to at once. Everything shares it: your instructions, the conversation so far, whatever you retrieved, the tool definitions, and the answer being generated.
The trap is reading a large number as a promise of comprehension. A model that accepts a very long input is telling you what will be accepted, not what will be used. Those are different claims, and only the first is guaranteed.
Two things degrade as the input grows, and neither is dramatic enough to look like a failure.
Position matters. Information at the beginning and end of a long input is used more reliably than information in the middle. Bury the one line that matters halfway through a long document and the model may answer as though it were not there. Not always. Often enough to design around.
Competition matters. Attention distributes a fixed amount of weight. Adding twenty more paragraphs does not give the relevant one more attention, it gives it a smaller share. Padding the prompt with plausibly-related material can make an answer worse while looking like diligence.
The benchmark that misleads
You will see claims of near-perfect recall over enormous inputs. Those usually come from a test that hides one distinctive sentence in a mass of unrelated text and asks the model to find it.
That test measures the easy case. The planted sentence has nothing competing with it, so it stands out. Real inputs are the opposite: forty passages that are all about the subject, one of which is right and current, and the others plausible, stale, or contradictory. Distinguishing among similar candidates is much harder than spotting an outlier, and it is what your system will actually be asked to do.
So when you read a recall claim, ask what the distractors looked like. If they were unrelated, the number does not transfer.
The decision this sets up
Every context question reduces to the same trade, and you now have both sides of it.
More context buys coverage: a lower chance the answer is missing entirely. It costs money and latency on a quadratic curve, and it dilutes attention across more material.
Less context buys precision and speed, and risks omitting the thing that mattered.
The resolution is not a number, it is a strategy: spend the budget on relevance rather than volume. Put fewer, better-selected things in the window. That sentence is the entire argument for retrieval, and module 4 is how you do it.
Two habits worth forming now:
- Put the most important material at the start or the end, never buried in the middle of a long block.
- Treat every addition as spending. If a paragraph is not likely to change the answer, it is costing you money, latency, and attention share for nothing.
Do this before moving on
Take a document of a few thousand words with a specific fact in it. Ask the model a question that fact answers, three times: with the fact near the beginning, near the end, and buried in the middle.
Then do a fourth run with only the relevant paragraph and nothing else. Compare all four for accuracy, latency, and rough cost. The fourth run is usually the best answer on every axis, and that result is the reason the next two modules exist.
Go deeper
- Context window is the reference for what shares the budget and why the limit exists at all.
- Context rot is the detailed version of the degradation above, including why the needle-in-a-haystack test overstates capability. Read it if this lesson interested you.
- Retrieval versus long context is the decision this lesson sets up, argued properly. It is the bridge into module 4.
- Practice question: When would you use a long context window instead of RAG? is a standard system-design probe, and "it depends" scores nothing without the cost curve.
- Practice question: How do you evaluate a long-context system? is where the benchmark critique above becomes your problem to solve.
Key takeaways
- The window says what fits, not what will be used well. Capacity is not recall.
- Material in the middle of a long input is used least reliably; put what matters at the start or the end.
- Attention is shared, so padding the prompt reduces the share going to the passage that mattered.
- Spend the context budget on relevance, not volume. That is the argument for retrieval in one line.
Check yourself before an interviewer does. Answer from memory first.
A summarisation feature misses a clause that is definitely in the document. Where is that clause most likely to be?