transformers

Almost everyone can quote softmax(QKᵀ/√d_k)V. The interviewer wants the variance argument and the exact training failure the scale prevents. Here is the answer that separates memorization from understanding.

A clean fundamentals question that catches people who only know GPT. The signal is matching the architecture to the task by its attention pattern, and explaining why the field consolidated on decoder-only even for non-generation tasks. Here is that answer.

Cross-attention is how a model attends from one sequence into another, the mechanism behind encoder-decoder models and multimodal conditioning. The signal is knowing exactly where the queries vs keys/values come from.

Everyone fixates on attention, but the feed-forward layers hold most of a transformer's weights and do much of the per-token 'knowledge' work. The signal is knowing what the FFN computes and why it dominates the parameter count.

Both are conditional compute, but one scales parameters and the other scales depth. The signal is knowing which axis each routes along and why that changes the FLOP story. Here is the sharp version.

The from-scratch implementation that frontier labs and NVIDIA actually ask. Writing it proves you understand shapes, the scale, the causal mask, and how heads split and recombine. Here is a correct, readable implementation and the follow-ups.

Single-head attention is common; multi-head adds the split-into-heads, attend-per-head, concatenate structure. The signal is the reshape into heads and a clear reason why multiple heads help. Here is the implementation.

RoPE is why modern LLMs extrapolate to longer contexts, and it's a rotation, not an addition. Coding it shows you actually understand how position enters attention. Here is the implementation.