kv cache
Applied AI interview questions tagged kv cache, across every topic.
14 questions · 2 unlocked for you
Concepts behind "kv cache"
The curriculum that explains the ideas these questions test.
Foundational
The Context WindowThe context window is the maximum number of tokens a model can attend to at once, prompt plus generation. It is bounded by attention's quadratic cost, the KV cache's linear memory growth, and the length the model was trained on. A bigger window is not free or uniformly useful (models lose information in the middle), which is why retrieval often beats stuffing everything into context. Applied-AI interviews probe it because it shapes cost, latency, and the RAG-vs-long-context decision.🧠 Foundations of LLMs & GenAI
Foundational
GPU Memory and the Serving StackServing an LLM is mostly a memory problem: the GPU must hold the model weights plus a KV cache that grows with sequence length and batch size, and inference splits into a compute-bound prefill and a memory-bandwidth-bound decode. Knowing the memory math (weights plus KV cache), why decode is bandwidth-bound, and the levers (quantization, batching, paged attention) is the foundation of LLM serving. Applied-AI interviews probe it because 'will this model fit and how fast will it run?' is a constant production question.🖥️ ML Infrastructure & Serving
Core
Attention and Self-AttentionAttention turns each token into a query, key, and value, scores every query against every key, softmaxes those scores into weights, and returns the weighted sum of values, so each token pulls in information from the others. Self-attention does this within one sequence. The all-pairs scoring is why cost grows with the square of sequence length, which in turn explains context limits, long-prompt expense, and the KV cache. Applied-AI interviews probe it because it links architecture to cost and latency in one mental model.🧠 Foundations of LLMs & GenAISign in
Core
Attention Variants: MHA, MQA, and GQAMulti-head attention gives every query head its own key and value heads, which is expressive but makes the KV cache large and memory-bandwidth hungry at decode time. Multi-query attention shares one key-value head across all query heads to shrink the cache hard, and grouped-query attention sits in between by sharing key-value heads across small groups. Applied-AI interviews probe this because it is the cleanest example of trading model quality against serving memory and throughput, and it explains why frontier models standardized on GQA.🧠 Foundations of LLMs & GenAISign in
Core
The KV CacheDuring autoregressive decoding, a model would recompute attention over the entire history at every step; the KV cache stores each token's key and value vectors so each new token only attends, never recomputes. The win is compute; the cost moves to memory: the cache grows with sequence length times batch size and usually becomes the binding constraint in serving. Applied-AI interviews probe it because it explains why long contexts are expensive to serve, why throughput (not model speed) is often the limit, and why MQA/GQA and PagedAttention exist.🧠 Foundations of LLMs & GenAISign in
Advanced
Disaggregated Prefill/Decode and Prefix CachingLLM inference has two phases with opposite hardware profiles: prefill is compute-bound (it processes the whole prompt in parallel) while decode is memory-bandwidth bound (one token at a time). Running both on the same GPU pool makes them fight, so long prefills stall ongoing decodes and you miss either the time-to-first-token or the time-per-output-token SLO. Disaggregation runs them on separate GPU pools and transfers the KV cache between them, and prefix caching reuses KV for shared prompt prefixes. Applied-AI interviews probe it because it is the current frontier of serving architecture and a real latency-SLO tradeoff.🖥️ ML Infrastructure & Serving🔒 Premium