memory bandwidth

Picking GPUs is a real applied decision, and 'get the biggest one' misses the point. The signal is knowing which spec binds your workload (VRAM, bandwidth, interconnect) and the train-versus-serve difference that flips the answer.

The counterintuitive truth of LLM serving: token generation is limited by how fast you can read weights from memory, not by math. Once you see that, the whole optimization menu falls out of one number.