memory
Applied AI interview questions tagged memory, across every topic.
8 questions · 0 unlocked for you
Concepts behind "memory"
The curriculum that explains the ideas these questions test.
Foundational
GPU Memory and the Serving StackServing an LLM is mostly a memory problem: the GPU must hold the model weights plus a KV cache that grows with sequence length and batch size, and inference splits into a compute-bound prefill and a memory-bandwidth-bound decode. Knowing the memory math (weights plus KV cache), why decode is bandwidth-bound, and the levers (quantization, batching, paged attention) is the foundation of LLM serving. Applied-AI interviews probe it because 'will this model fit and how fast will it run?' is a constant production question.🖥️ ML Infrastructure & Serving
Core
Agent Memory: Short-Term, Long-Term, and Memory StoresAgent memory is how an agent retains and recalls information across steps and sessions. Short-term (working) memory is what lives in the context window for the current task; long-term memory is durable information (facts, user preferences, past outcomes) stored outside the window and retrieved when relevant. The skill is deciding what is worth remembering, where to store it, and when to read it back. Applied AI interviews probe it because durable memory is what turns a one-shot chatbot into an agent that improves over time.🤖 Retrieval & AgentsSign in
Core
Context Engineering for AgentsContext engineering is the discipline of designing the full information payload that goes into an agent's context window each turn: system instructions, memory, retrieved data, tool definitions and results, and conversation history. Most agent failures are context failures, where the right information is absent, buried, stale, or crowding out the rest of the budget. Applied AI interviews probe it because it is the highest-leverage lever on agent reliability and cost, and it separates people who tune prompts from people who manage state.🤖 Retrieval & AgentsSign in
Core
Quantization and Low PrecisionQuantization stores and computes model weights (and activations) in fewer bits, FP16/BF16, FP8, INT8, INT4, instead of FP32, cutting memory and speeding inference at some accuracy cost. It is the main lever to fit a large model on a given GPU and to serve it cheaply, and it underlies QLoRA fine-tuning and KV-cache compression. Applied-AI interviews probe it because 'how do you serve a 70B model affordably?' usually starts with quantization, and knowing the precision ladder and its trade-offs is essential.🖥️ ML Infrastructure & ServingSign in
Core
PagedAttentionThe KV cache is the memory bottleneck in LLM serving, and naively reserving a contiguous block per request (sized for the maximum length) wastes most of it to fragmentation and over-allocation. PagedAttention borrows virtual-memory paging: store the KV cache in fixed-size non-contiguous pages allocated on demand, so memory is used only as tokens are generated. This packs far more concurrent requests onto a GPU, raising throughput. Applied-AI interviews probe it because it is the key memory innovation behind modern serving (vLLM).🖥️ ML Infrastructure & ServingSign in
Core
Streaming and BackpressureWhen data is too big to fit in memory or arrives continuously, you process it as a stream, one piece at a time, with bounded memory, rather than loading it all. Backpressure is the mechanism that stops a fast producer from overwhelming a slow consumer, by signaling 'slow down' rather than buffering unboundedly until you run out of memory. Applied-AI interviews probe it because AI pipelines process huge datasets and token streams, and the naive load-everything approach OOMs while unbounded buffering crashes under load.💻 Coding & Engineering CraftSign in