TL;DR: Knowledge distillation trains a small student to mimic a large teacher, using the teacher's soft probability distribution (not just hard labels) as the target, which carries the "dark knowledge" of how classes relate. You get a much smaller, faster model that keeps most of the teacher's quality. It is one of three compression levers: distillation (train a new smaller model), quantization (fewer bits per weight), and pruning (remove weights or structures). They stack, so the real question is the order.
How to approach it. Define distillation in one line (student mimics teacher), then land the key insight: soft targets transfer more than hard labels. Then place it against quantization and pruning so you show command of the whole toolkit and when each fits. The interviewer is screening for the why soft targets and the versus other methods, not a textbook definition.
A strong answer. How distillation works. You train a small student to reproduce a large teacher's behavior. The trick is the target: instead of (or alongside) the hard ground-truth labels, the student learns from the teacher's soft output distribution, the full probability vector, usually softened with a temperature. Those soft targets carry dark knowledge: the probabilities the teacher puts on the wrong classes encode how similar classes are. An image of a cat gets some mass on "tiger" and near-zero on "truck," which is far richer signal than a one-hot label. So the student learns the teacher's generalization, not just the answers, and reaches accuracy a same-size model trained from scratch on hard labels usually cannot. For LLMs, you distill a large model's outputs or reasoning traces into a smaller one to cut serving cost.
When to use it. You have a high-quality but expensive model and need something smaller and faster for serving: latency, cost, edge or on-device. Distillation gives the best quality-per-parameter of the three options because the small model is trained to match the big one, not just shrunk after the fact. The price is a training run plus access to the teacher or its outputs.
Versus the other compression levers:
| Lever | What it does | Retraining | Best for |
|---|---|---|---|
| Quantization | Lower precision (FP16 to INT8/INT4) | None (PTQ) | Cheap first win, big memory/speed gain |
| Pruning | Remove weights or structures | Usually fine-tune | More headroom when you can retrain |
| Distillation | Train a new smaller architecture | Full training run | A genuinely smaller model that keeps quality |
Quantization is the easiest first lever: quality holds well at INT8 and degrades at very low bits. Unstructured pruning zeros individual weights (needs sparse-kernel support to actually speed up); structured pruning removes whole neurons, heads, or channels and gives real speedup on standard hardware. Distillation is the most flexible because you choose the student's size and shape, and the most work.
They are complementary and stacked: distill to a smaller model, then quantize it (and possibly prune) for serving.
The defensible order: quantize first (cheap, no retrain), prune if you need more and can fine-tune, distill when you want a fundamentally smaller model that keeps quality and can afford to train it.
Key takeaways.
- Soft targets carry inter-class similarity (dark knowledge); that is why distillation beats from-scratch training at the same size.
- Distillation trains a new model; quantization and pruning shrink an existing one.
- The three stack: distill, then quantize, then prune for serving.
- Default order is quantize, then prune, then distill, by cost of applying.
What interviewers probe next.
- "Why do soft targets help more than hard labels?" They encode inter-class similarity and give a smoother, more informative signal than one-hot, so the student inherits the teacher's generalization.
- "Distillation versus just training a small model on the data?" The teacher's soft targets transfer knowledge the small model could not easily learn from raw labels, so it usually beats from-scratch at that size.
- "Pick one to start with?" Quantization: cheapest, no retraining, big wins. Reach for distillation when you need a fundamentally smaller model and can train it.
- "Self-distillation or data-free distillation?" Variants exist (student same size as teacher, or distilling on synthetic inputs without the original data). Mention them if relevant.
Common mistakes.
- Saying distillation "compresses the model" without the soft-target dark-knowledge insight that makes it work.
- Confusing it with quantization or pruning; distillation trains a new model, the others shrink an existing one.
- Forgetting distillation needs a training run and teacher access, unlike post-training quantization.
- Treating the three as alternatives rather than complementary levers that stack.