attention
Applied AI interview questions tagged attention, across every topic.
13 questions · 5 unlocked for you
Concepts behind "attention"
The curriculum that explains the ideas these questions test.
Foundational
From RNNs to Transformers: RNN, LSTM, Seq2SeqRecurrent networks process sequences one step at a time through a hidden state, which makes them principled but slow and bad at long-range dependencies because gradients vanish across many steps. LSTMs and GRUs add gates to carry information further, and seq2seq encoder-decoder models with attention removed the single-vector bottleneck, which is the idea transformers then took to its conclusion. Applied-AI interviews probe this because it explains why attention exists and why we abandoned recurrence for parallelism.🧠 Foundations of LLMs & GenAI
Foundational
The Context WindowThe context window is the maximum number of tokens a model can attend to at once, prompt plus generation. It is bounded by attention's quadratic cost, the KV cache's linear memory growth, and the length the model was trained on. A bigger window is not free or uniformly useful (models lose information in the middle), which is why retrieval often beats stuffing everything into context. Applied-AI interviews probe it because it shapes cost, latency, and the RAG-vs-long-context decision.🧠 Foundations of LLMs & GenAI
Core
The Transformer ArchitectureThe transformer is the architecture behind modern LLMs: stacked blocks that each mix information across tokens with self-attention and then transform each token with a feed-forward network, wrapped in residual connections and normalization. Understanding the two sub-layers (attention mixes across tokens, the feed-forward processes each one) explains where parameters live, why Mixture-of-Experts scales the feed-forward, and why decoder-only models dominate. Applied-AI interviews probe it because it is the mental scaffold for everything else, attention cost, KV cache, MoE, and serving.🧠 Foundations of LLMs & GenAISign in
Core
Attention and Self-AttentionAttention turns each token into a query, key, and value, scores every query against every key, softmaxes those scores into weights, and returns the weighted sum of values, so each token pulls in information from the others. Self-attention does this within one sequence. The all-pairs scoring is why cost grows with the square of sequence length, which in turn explains context limits, long-prompt expense, and the KV cache. Applied-AI interviews probe it because it links architecture to cost and latency in one mental model.🧠 Foundations of LLMs & GenAISign in
Core
FlashAttention and IO-Aware KernelsNaive attention is slow not because of the matmuls but because it writes the full N-by-N attention matrix to GPU high-bandwidth memory and reads it back, which is memory-bandwidth bound. FlashAttention fuses the whole attention computation into one kernel that tiles the inputs in fast on-chip SRAM and never materializes the full matrix, using an online-softmax trick to stay exact. Applied-AI interviews probe it because it is why long-context training and serving became affordable and a clean test of GPU memory-hierarchy reasoning.🖥️ ML Infrastructure & ServingSign in