← 🖥️ ML Infrastructure & ServingNEXT IN ML INFRASTRUCTURE & SERVINGDisaggregated Prefill/Decode and Prefix Caching→
Core
PagedAttention
The KV cache is the memory bottleneck in LLM serving, and naively reserving a contiguous block per request (sized for the maximum length) wastes most of it to fragmentation and over-allocation. PagedAttention borrows virtual-memory paging: store the KV cache in fixed-size non-contiguous pages allocated on demand, so memory is used only as tokens are generated. This packs far more concurrent requests onto a GPU, raising throughput. Applied-AI interviews probe it because it is the key memory innovation behind modern serving (vLLM).
a free account unlocks the core curriculum tier · no card
RELATED CONCEPTS
PRACTICE THIS IN REAL QUESTIONS
ML Infrastructure & GPUsHow do you offload the KV cache to CPU or NVMe, and when is it worth the bandwidth hit?→LLM & GenAI FundamentalsWhat is PagedAttention, and why did it transform LLM serving throughput?→ML Infrastructure & GPUsServe a 70B-parameter model with high throughput. Do the memory math and name the optimizations.→ML Infrastructure & GPUsExplain the KV cache: prefill vs decode, why it grows, and how MQA/GQA and PagedAttention help.→ML Infrastructure & GPUsYour inference server OOMs when requests arrive with long prompts. How do you handle variable-length memory?→ML Infrastructure & GPUsHow does prefix caching work internally in an LLM server, and when does it actually help?→
COMPANIES THAT ASSUME THIS