21How do you profile and diagnose LLM inference performance (TTFT, inter-token latency, GPU utilization)?▼medium★ EssentialNVIDIAMicrosoftOpenAI1 replies◆ premiumLLM serving has its own metrics, and one latency number hides the real bottleneck. The signal is splitting prefill from decode and reading GPU utilization as a clue, not a verdict. Here is the diagnostic toolkit.Open full answer →
28What is MFU (Model FLOPs Utilization), and why can GPU utilization be misleading?▼hard★ EssentialNVIDIAOpenAIGoogle1 replies◆ premiumnvidia-smi showing 100% can hide that you are using a fraction of the hardware's real compute. The signal is MFU (useful FLOPs vs peak) and the gap between 'the GPU is busy' and 'the GPU is efficient'.Open full answer →
