18What are the collective communication operations (all-reduce, all-gather, reduce-scatter) in distributed training?▼hardNVIDIAOpenAIGoogle2 replies○ sign inDistributed training is bottlenecked by GPU-to-GPU communication, and these collectives are how the data moves. The signal is what each one does and which parallelism strategy depends on it. Here is the answer.Open full answer →
58How do you overlap communication with computation in distributed training, and how do you verify it works?▼hardNVIDIAMetaGoogle1 replies◆ premiumThe collective communication in distributed training is pure overhead unless it runs while the GPU computes. Hiding it is the difference between 30% and 55% MFU. Here is how the overlap actually works and how you confirm it on a trace.Open full answer →
