training

Distillation buys you most of a frontier model's quality at a fraction of the serving cost. The signal is naming the three LLM-specific variants and knowing which one works against a closed API. Here is the answer.

Distillation is not one method. The signal is knowing when to match logits versus train on generated text, why on-policy distillation beats off-policy, and how reasoning models are distilled.

The ratio of web, code, books, and math in pretraining quietly decides downstream skills. The signal is knowing how mixtures are chosen, why upsampling helps, and how methods like DoReMi automate it.

Continued pretraining injects domain knowledge that fine-tuning cannot, but naively it wrecks general ability. The signal is the replay ratio, learning-rate rewarming, and how you measure forgetting.

FP8 is the format behind modern H100-class training and serving. The signal is knowing the two FP8 variants, why a floating format beats INT8 for dynamic range, and where scaling still matters.

Many state-of-the-art training runs keep a shadow copy of the weights that's a moving average of the trajectory, and serve that instead of the final step. It's a few lines and a real accuracy boost. Here it is.

Almost every modern training run uses warmup-then-cosine, and getting it wrong destabilizes early training or wastes the tail. It's a closed-form function of the step. Here is the implementation and the why.

Optimizer questions test whether you understand what each one adapts and the subtle AdamW fix that the whole field now uses. The signal is the per-parameter adaptivity story plus why coupling weight decay to Adam was a bug. Here is that answer.

Backprop is the algorithm that makes deep learning trainable, and the interviewer wants the chain-rule mechanics, not just 'it computes gradients.' The signal is the forward-then-backward flow and why it is efficient. Here is the answer.

BatchNorm is one of the most-asked deep-learning questions, and the trap is the train/inference difference. The signal is what it normalizes, why it stabilizes and speeds training, and why it switches to running statistics at inference. Here is that answer.

The learning rate is the one hyperparameter that can NaN your run on step 50 or stall it forever. Modern training schedules it instead of fixing it. The signal is knowing exactly what warmup and decay each buy you. Here is the answer.

'The model won't train' has a short list of usual suspects, each with a distinctive loss-curve signature. The shape of the curve names the bug before you touch a single hyperparameter. Here is how to read it.

The idea that models, like students, learn better from easy examples first is intuitive, and sometimes it works and sometimes it does nothing. Knowing when it pays off is the real signal. Here is the honest answer.

Mixed precision is standard at scale, and the interviewer wants the numerics: why FP16 needs loss scaling, why BF16 mostly does not, and what stays in FP32. The signal is understanding dynamic range vs precision. Here is that answer.

Activations, not just weights, can dominate training memory, and gradient checkpointing is the standard fix. The signal is the precise trade: recompute activations in the backward pass instead of storing them. Here is the answer.

Batch size is a training knob with non-obvious effects on speed, memory, and generalization. The signal is the large-batch tradeoffs and the learning-rate-scaling and gradient-accumulation tricks that make scaling actually work.

Beyond FP16/BF16, FP8 is the next rung on the precision ladder, and the latest tensor cores compute on it natively. The signal is knowing the two formats, what FP8 actually buys, and why dynamic range forces careful scaling.

Paying for accelerators that idle half the time is the most common waste in ML training, and the instinct to add more GPUs makes it strictly worse. The interviewer wants the profiling discipline that finds what is starving them.

At billion-parameter scale, training can be cruising and then the loss jumps and never comes back, burning a fortune in compute. The causes and the playbook are well-known to the few who've done it. Here it is.

Full activation checkpointing saves memory but burns a flat 30% extra compute. Selective recomputation recovers most of that by only recomputing the cheap, memory-heavy operations. Here is how to pick what to recompute.

FP8 promises to roughly double training throughput over BF16, but with only a handful of mantissa bits the numerics are unforgiving. Per-tensor scaling and a selective recipe are what make it converge.