TL;DR: Quantization maps weights/activations from FP16 to lower-precision integers (INT8, INT4), shrinking memory and speeding memory-bound inference. Naive rounding fails because a few large-magnitude outliers blow up the scale; methods like GPTQ and AWQ handle this (error-compensated rounding, protecting salient weights). Weight-only quantization is safe and common; activation quantization is harder. Always validate on your eval set, not just perplexity.
How to approach it
State why you quantize (memory to fit, and bandwidth-bound decode speedups), then the central difficulty (dynamic range and outliers), then the methods and the validation step, because "did quality actually hold" is the real question. The candidates who pass name outliers as the failure mode and name a task eval, not perplexity, as the proof.
A strong answer
Why. Weights at FP16 are 2 bytes each; INT8 halves that, INT4 quarters it. Beyond fit, LLM decode is memory-bandwidth-bound (you stream weights per token), so reading fewer bytes per weight directly speeds generation. That is why a quantized 70B can serve on one GPU and run faster.
The hard part: dynamic range. Quantization maps a float range to a small integer grid via a scale. If a tensor has a few huge-magnitude outliers, the scale stretches to cover them and the many normal values collapse into a couple of buckets, destroying precision. Naive round-to-nearest at INT4 wrecks accuracy for exactly this reason.
Methods.
- Weight-only PTQ (GPTQ, AWQ). GPTQ quantizes weights greedily while compensating for the error introduced, using second-order (Hessian) information; AWQ identifies and protects the salient weight channels that matter most for output. Both are post-training (no retraining), need only a small calibration set, and hold quality well at INT4 for many models.
- Activation quantization (INT8 weights+activations, SmoothQuant) is harder because activation outliers are large and input-dependent; SmoothQuant shifts difficulty from activations into weights to make it tractable.
- FP8 (on newer hardware) keeps floating-point dynamic range at 8 bits and is increasingly used for both training and inference.
- QLoRA is the training-side cousin: fine-tune adapters on top of a 4-bit frozen base.
| Precision | Quality | When to use |
|---|---|---|
| INT8 weight-only | Near-lossless for most models | Safe default for memory and speed |
| INT4 (AWQ/GPTQ) | Small hit, usually fine | Fit a big model on one GPU, validated on tasks |
| INT8 weights+activations | Trickier, outlier-sensitive | Only with SmoothQuant and measurement |
| Below INT4 | Degrades fast | Only with QAT |
Validation. Perplexity alone is a weak proxy; run your task eval set (accuracy, exact-match, judge scores) and compare to the FP16 baseline. Quantization error is non-uniform across capabilities, so a model can hold perplexity yet regress on reasoning or code. Keep sensitive components (sometimes the embedding/output layers, or attention) at higher precision if needed. The defensible position: weight-only INT4 (AWQ/GPTQ) is a safe default for serving, validated on real tasks; push to activation or INT4-everything only with measurement.
Key takeaways
- The failure mode is outliers stretching the scale, not "fewer bits" in the abstract; GPTQ/AWQ exist to fix it.
- INT8 is near-lossless; INT4 weight-only is the workhorse serving default; below INT4 needs QAT.
- Activation quantization is strictly harder than weight-only because activation outliers are input-dependent.
- Validate on a task eval against the FP16 baseline, not perplexity alone, because errors hit capabilities unevenly.
What interviewers probe next
- "Why does INT4 help latency, not just memory?" Decode is bandwidth-bound; fewer bytes per weight means less memory traffic per token.
- "Per-tensor vs per-channel/group scales?" Finer-grained scales (per-channel, group-wise) handle varying ranges far better than one global scale, at a small overhead.
- "QAT vs PTQ?" Quantization-aware training recovers more accuracy at very low bits but costs a training run; PTQ (GPTQ/AWQ) is cheaper and usually enough at INT8/INT4.
- "How low can you go?" INT8 is nearly lossless for most models; INT4 is usually fine with good methods; below that quality degrades fast without QAT.
Common mistakes
- "Just use fewer bits" with no mention of outliers or dynamic range, the thing that actually breaks.
- Validating on perplexity only and missing task-specific regressions.
- Treating activation quantization as easy as weight quantization.
- Calibrating on unrepresentative data, so the scales are wrong for production inputs.