llama cpp

GGUF is the format behind most local LLM runs, and its quant names look cryptic. The signal is decoding what Q4_K_M means and why mixed-precision k-quants beat naive uniform quantization.

On-device AI is a real product surface (privacy, offline, latency), and it forces hard constraints. The signal is the quantization plus format plus runtime stack and the tradeoffs you accept under tight memory and battery budgets.