quantization

Fine-tuning a 65B model used to mean a node of A100s. QLoRA collapses it onto one card with three specific tricks. The signal is knowing what gets quantized, what stays trainable, and why quality barely moves. Here is the answer.

At long context the KV cache, not the weights, fills the GPU. The signal is naming the levers (quantization, token eviction, head sharing, low-rank) and what each one costs in quality.

Both squeeze an LLM to 4-bit weights, but they decide what to protect very differently. The signal is GPTQ's error-correcting solve versus AWQ's activation-aware scaling, and the calibration each needs.

GGUF is the format behind most local LLM runs, and its quant names look cryptic. The signal is decoding what Q4_K_M means and why mixed-precision k-quants beat naive uniform quantization.

FP8 is the format behind modern H100-class training and serving. The signal is knowing the two FP8 variants, why a floating format beats INT8 for dynamic range, and where scaling still matters.

At a billion vectors the index choice is a memory budget problem before it's a recall problem. Flat search is out, HNSW may not fit, and PQ trades recall for RAM. Here is the decision a staff engineer makes on the whiteboard.

On-device AI trades raw capability for privacy, offline use, and latency, under brutal memory and battery limits. The signal is the small-quantized-model stack plus a hybrid router that escalates the hard queries to the cloud without leaking private context.

The trap is jumping straight to 'add more GPUs.' The signal is profiling first, then applying the cheap, no-retrain levers in the right order. Here is the diagnosis-then-optimize playbook for p99.

The interviewer wants real numbers, not 'use a big GPU.' Weights are fixed, the KV cache grows with load, and the lever order decides everything. Here is the back-of-envelope and the serving stack.

Quantization is the first lever for fitting and speeding up models, and the interviewer wants more than 'use fewer bits.' The signal is knowing what precision buys you, why outliers break naive quantization, and how you prove quality held. Here is that answer.

Distillation buys you a small fast model that keeps most of a big model's quality. The signal is explaining why soft targets transfer more than hard labels, and knowing when to distill versus quantize versus prune. Here is the model-compression answer.

On-device AI is a real product surface (privacy, offline, latency), and it forces hard constraints. The signal is the quantization plus format plus runtime stack and the tradeoffs you accept under tight memory and battery budgets.

Beyond FP16/BF16, FP8 is the next rung on the precision ladder, and the latest tensor cores compute on it natively. The signal is knowing the two formats, what FP8 actually buys, and why dynamic range forces careful scaling.

At long context the KV cache, not the weights, dominates GPU memory and decode bandwidth. The signal is recognizing it as the binding constraint and naming the levers that shrink it without wrecking quality.

Pruning removes redundant weights to shrink and sometimes speed up a model. The signal is the structured-versus-unstructured split and why only structured pruning reliably wins latency on commodity hardware.

Naive 4-bit quantization can tank quality, and the reflex to give up and serve fp16 leaves a big speedup on the table. The accuracy is usually recoverable. Here is the ladder of fixes.

FP8 serving is fast and usually fine, until it silently degrades on the one workload your benchmark did not cover. The signal is knowing what FP8 breaks and how to validate it. Here is the recipe.