TL;DR: Long context wins when the relevant material is small enough to fit, fits the budget, and you want simplicity; RAG wins when the corpus is large, changes often, needs citations, or cost/latency matter. Long context is not free: you pay per token on every call, latency and KV-cache grow with length, and models attend unevenly across a long prompt ("lost in the middle"). In practice many systems do both: retrieve to narrow, then use a generous context window.
How to approach it. Reject the framing that one killed the other. Lay out the axes that actually decide it (corpus size, freshness, cost/latency, accuracy at length, need for citations), give a default, and name the long-context failure mode the interviewer is listening for.
A strong answer. The choice is one decision across a handful of axes, and each axis pushes a clear direction.
| Axis | Pushes to long context | Pushes to RAG |
|---|---|---|
| Corpus size | Fits the window (a few docs) | Millions of docs, never fits |
| Cost at scale | Low QPS, small context | Pay per input token every call |
| Latency / memory | Short prompts | Long prefill, KV-cache caps concurrency |
| Freshness | Static material | Changes often, re-index without resend |
| Accuracy at length | Relevant info is short | Mid-context info gets lost |
| Citations | Not required | Source attribution needed |
A few of these deserve the detail an interviewer wants:
- Cost. You pay per input token on every request. Feeding 100k tokens of context on each call is expensive at scale; RAG sends only the few retrieved chunks, often orders of magnitude fewer tokens. At high QPS this dominates the decision.
- Latency and memory. Prefill cost and KV-cache memory grow with prompt length, so a giant context raises time-to-first-token and caps concurrency. RAG keeps prompts short.
- Accuracy at length ("lost in the middle"). Models do not attend uniformly across a long prompt: performance is typically highest when relevant information sits near the beginning or end and degrades when it lands in the middle (the documented U-shaped curve, Liu et al. 2023). A huge context is no guarantee the model uses the buried relevant part. Retrieval that places a few highly relevant chunks prominently can beat dumping everything.
Default: for a small, stable, self-contained set of documents per request, long context is simpler and fine. For a large, changing, or cost-sensitive corpus, use RAG. The common production answer is hybrid: retrieve to narrow the corpus to what matters, then use a comfortably large context window so you are not over-chunking. You get retrieval's cost, freshness, and citations with long context's tolerance for less aggressive splitting.
Key takeaways
- Big windows changed the chunking tradeoff, not the need to retrieve from large or changing corpora.
- Long context bills the full window on every call; RAG bills only retrieved tokens, which decides it at high QPS.
- "Lost in the middle" means a fact technically present in context can still be missed, so placement beats volume.
- The production default is hybrid: retrieve to narrow, then fill a generous context window.
What interviewers probe next.
- "What is 'lost in the middle'?" Long-context models attend best to the start and end of the prompt and worse to the middle, so relevant info buried mid-context can be missed even though it is technically present.
- "Cost math?" Per-call input tokens times price times QPS; long context multiplies that by the full window every request, RAG by only the retrieved tokens.
- "Does a bigger window kill RAG?" No: scale, cost, freshness, and citations still favor retrieval for large/changing corpora; the window size changes the chunking tradeoff, not the need to retrieve.
- "How do you decide the split in a hybrid system?" Retrieve enough to ensure recall, then size the context to comfortably hold the top results plus margin, tuned against eval metrics and cost.
Common mistakes.
- Declaring RAG obsolete because context windows grew, ignoring cost, scale, freshness, and citations.
- Assuming everything in a long context is used equally, missing lost-in-the-middle.
- Ignoring the per-call token cost and latency of stuffing huge contexts at scale.
- Treating it as binary instead of the common retrieve-then-long-context hybrid.