long context

Context windows are not arbitrary numbers; they fall out of attention's quadratic cost, KV-cache memory, and how the model was trained. The signal is naming all three causes and matching each to the technique that extends it.

Full attention is O(n^2), so long context needs cheaper attention. The signal is naming the families (sparse, sliding-window, linear), what each gives up, and why FlashAttention is not one of them.

Modern LLMs dropped learned absolute positions for RoPE and ALiBi to handle long context. The signal is knowing both encode RELATIVE position, and exactly how that enables length extrapolation past the trained window.

A model advertising a million-token window may not use it. The signal is knowing needle-in-a-haystack, where it breaks, and how to test real multi-fact reasoning across the full context. Here is the answer.

Drop the oldest tokens to bound the KV cache and the model collapses. Keep just the first few tokens and it generates indefinitely. The signal is understanding why those first tokens act as an attention sink.

A model trained at 4K tokens can be stretched to 128K without full retraining. The signal is knowing why naive extrapolation fails and how PI, NTK-aware scaling, and YaRN each rescale RoPE frequencies differently.

At long context the KV cache, not the weights, fills the GPU. The signal is naming the levers (quantization, token eviction, head sharing, low-rank) and what each one costs in quality.

A 2025-2026 question that catches people clinging to dogma in either direction. The signal is a cost, latency, accuracy, and scale tradeoff, plus knowing the 'lost in the middle' failure of long context. Here is the framework for choosing.

Late chunking flips the usual order of operations to fix the lost-context problem without a single extra LLM call. The whole signal is which step happens first. Here is the answer.

A VLM that reads one page perfectly can still fail an 80-page contract, and the reason is arithmetic before it is model quality. The signal is picking the right architecture and knowing what per-page processing silently loses. Here is the answer.

Two models chained (ASR then summarization), a long-context problem hiding inside, and a faithfulness bar where a fabricated action item has real consequences. The signal is the pipeline plus how you keep a 3-hour transcript honest at thousands a day.

Two forces decide this design: documents longer than the context window, and summaries that must not lie. The strong answer pairs hierarchical map-reduce with claim-level grounding, then serves it cheaply at scale.

Summarizing a 200-page contract is not one LLM call: it is chunking, hierarchical reduction, and a faithfulness check so you never invent facts. Learn the map-reduce pattern, when long-context beats it, and how to evaluate summaries at scale.

At long context the KV cache, not the weights, dominates GPU memory and decode bandwidth. The signal is recognizing it as the binding constraint and naming the levers that shrink it without wrecking quality.

Data, tensor, and pipeline parallelism all leave one sequence's activations on one device, so 200k+ token training hits a wall none of them can fix. The fourth axis shards the sequence itself, and the interview lives in the communication math.