18What are the collective communication operations (all-reduce, all-gather, reduce-scatter) in distributed training?▼hardNVIDIAOpenAIGoogle2 replies○ sign inDistributed training is bottlenecked by GPU-to-GPU communication, and these collectives are how the data moves. The signal is what each one does and which parallelism strategy depends on it. Here is the answer.Open full answer →
38Your distributed training job hangs or crashes intermittently. How do you debug it?▼hardMetaNVIDIAOpenAI2 replies◆ premiumA 256-GPU job that wedges with no error at 3am is a special kind of pain. The causes are a short, recurring list. Here is the systematic way to find which one bit you.Open full answer →
62Your training collectives are slow. How do you debug the NCCL/interconnect path and find where bandwidth is lost?▼hardNVIDIAMetaMicrosoft2 replies◆ premiumWhen all-reduce is the bottleneck, the cause is almost always a misconfigured path: traffic on the wrong link, a downed NIC, or a topology NCCL never discovered. Here is the systematic way to find the lost bandwidth.Open full answer →
