paged attention

Most GPU memory in naive LLM serving is wasted on KV-cache fragmentation. PagedAttention borrows virtual memory paging to reclaim it. The signal is explaining the fragmentation problem and how blocks fix it.

The KV cache is why LLM serving is hard, and the interviewer wants the mechanics: what it stores, why it limits concurrency, and the tricks that shrink it. Here is the answer that shows you understand decode-time economics.

A serving box that's stable on short prompts crashes the moment a 30k-token request lands, because KV-cache memory scales with sequence length times batch. Here is how to bound it without crashing.

A shared system prompt gets re-prefilled on every request unless the server remembers it. Prefix caching skips that work, but only when the blocks line up exactly. Here is the hashing and eviction machinery underneath.

GPU memory runs out long before you run out of conversations to cache. Offloading KV to CPU or NVMe buys capacity, but the interconnect can become the new bottleneck. Here is the bandwidth math.