← 🖥️ ML Infrastructure & ServingNEXT IN ML INFRASTRUCTURE & SERVINGSpeculative Decoding→
Advanced
Disaggregated Prefill/Decode and Prefix Caching
LLM inference has two phases with opposite hardware profiles: prefill is compute-bound (it processes the whole prompt in parallel) while decode is memory-bandwidth bound (one token at a time). Running both on the same GPU pool makes them fight, so long prefills stall ongoing decodes and you miss either the time-to-first-token or the time-per-output-token SLO. Disaggregation runs them on separate GPU pools and transfers the KV cache between them, and prefix caching reuses KV for shared prompt prefixes. Applied-AI interviews probe it because it is the current frontier of serving architecture and a real latency-SLO tradeoff.
Unlock the full curriculum — ₹2,000 / $25every concept + every answer · 6 months · no auto-renew
RELATED CONCEPTS
PRACTICE THIS IN REAL QUESTIONS
ML Infrastructure & GPUsHow does prefix caching work internally in an LLM server, and when does it actually help?→ML Infrastructure & GPUsHow do you offload the KV cache to CPU or NVMe, and when is it worth the bandwidth hit?→ML Infrastructure & GPUsServe a 70B-parameter model with high throughput. Do the memory math and name the optimizations.→ML Infrastructure & GPUsExplain the KV cache: prefill vs decode, why it grows, and how MQA/GQA and PagedAttention help.→ML Infrastructure & GPUsWhat is chunked prefill, and how does it stop long prompts from stalling decode?→ML Infrastructure & GPUsDesign the request router and scheduler for a multi-replica LLM serving cluster.→
COMPANIES THAT ASSUME THIS