37You doubled the GPUs but training barely got faster. Why doesn't distributed training scale linearly?▼hardMetaNVIDIAGoogle1 replies◆ premiumLinear scaling is the marketing number; the real curve bends early for reasons that are physics, not bugs. Here is where the speedup goes and how to claw it back.Open full answer →
55How do you combine data, tensor, and pipeline parallelism (3D parallelism) to train a 175B model?▼expertNVIDIAMicrosoftMeta1 replies◆ premiumNo single parallelism dimension trains a 175B model on its own. The art is layering three of them onto the right hardware so the expensive communication stays on the fast links. Here is how to size each axis.Open full answer →
