The Context Window

TL;DR: The context window is the token budget the model can see at once: the prompt, any retrieved or pasted material, and the tokens it generates all share it. Its size is limited by attention's quadratic cost, the linearly-growing KV cache, and the sequence length the model was trained on. Bigger is not automatically better: cost and latency rise, and models attend poorly to the middle of a long context, which is the core reason retrieval is often preferable to a giant prompt.

What shares the budget

Everything the model processes for a request lives in the window: system prompt, conversation history, retrieved documents, the user's question, and the model's own output as it generates. When the window fills, something must be dropped or truncated. Thinking of context as a fixed budget you allocate (rather than free space) is the mental model that prevents most production surprises.

Why it is bounded

Three forces cap the window:

1. Quadratic attention. Every token attends to every other, so compute and the attention matrix scale with the square of the length (see attention). Doubling the context roughly quadruples that work.
2. KV-cache memory. At inference, each token's keys and values are cached; this memory grows linearly with length (and batch) and often becomes the binding constraint before compute does (see the KV cache).
3. Trained length. A model learns positions up to some maximum; pushed beyond it, quality degrades because it has never "seen" those positions. Extending it (RoPE scaling, fine-tuning on longer sequences) is real work, not a config flag.

"Lost in the middle"

A bigger window does not mean the model uses all of it well. Models reliably attend to the beginning and end of a long context and lose information in the middle. So a 200k-token window does not make "paste everything" a good strategy: the relevant fact buried at position 100k may be effectively ignored.

This is the crux of the retrieval-vs-long-context decision (see that concept). Retrieval puts only the few most relevant chunks in the window, which is cheaper, lower-latency, and often more accurate than stuffing a long document the model will partly ignore, plus it updates without retraining.

Why interviewers probe this

The window is where cost, latency, and quality collide. A candidate who treats "we have a 1M-token model" as solving the knowledge problem misses that long context is expensive, slow, and unevenly used. The strong move is to frame context as a budget, name the three limits, and reach for retrieval when the corpus is large, changing, or only partially relevant, reserving long context for genuinely cohesive inputs.

Common misconceptions

• "A bigger window solves knowledge problems." It raises cost and latency and is unevenly used; retrieval is often better for large or changing corpora.
• "The window is just the prompt." Output tokens consume the same budget; long generations can crowd out the prompt.
• "Models use the whole context equally." They attend best to the ends and lose the middle ("lost in the middle").
• "Extending context is a setting." Past the trained length, quality drops without positional-encoding tricks and fine-tuning.

Key takeaways

• The context window is a fixed token budget shared by prompt, retrieved material, history, and output.
• It is bounded by quadratic attention, linear KV-cache memory, and the trained sequence length.
• Models lose information in the middle of long contexts, so bigger is not uniformly better.
• These limits are why retrieval frequently beats stuffing everything into a long prompt.