distributed training

A from-scratch test of distributed input pipelines. The signal is partitioning data across workers with no overlap and no gaps, epoch-consistent shuffling with a shared seed, and handling the uneven-last-batch problem. Here is the implementation.

At thousand-GPU scale, hardware failure is the norm, not the exception, and a run that cannot resume wastes weeks. The signal is checkpointing strategy, deterministic resume, and minimizing lost work. Here is the fault-tolerant design.

The interviewer wants you to know why a model that fits on no single GPU still trains, and to do the optimizer-state memory math that motivates sharding. Here is the parallelism taxonomy and the 16-bytes-per-parameter calculation.

FSDP and ZeRO let you train models too big for one GPU without splitting the compute. The signal is the ZeRO stages (what gets sharded at each level) and the communication you pay for it. Here is the answer.

Distributed training is bottlenecked by GPU-to-GPU communication, and these collectives are how the data moves. The signal is what each one does and which parallelism strategy depends on it. Here is the answer.

Linear scaling is the marketing number; the real curve bends early for reasons that are physics, not bugs. Here is where the speedup goes and how to claw it back.

A 256-GPU job that wedges with no error at 3am is a special kind of pain. The causes are a short, recurring list. Here is the systematic way to find which one bit you.

No single parallelism dimension trains a 175B model on its own. The art is layering three of them onto the right hardware so the expensive communication stays on the fast links. Here is how to size each axis.

ZeRO and FSDP both shard training state across data-parallel ranks, but the magic is in when each shard is gathered and freed. Get the memory accounting and the all-gather timing and you can size any run.

The collective communication in distributed training is pure overhead unless it runs while the GPU computes. Hiding it is the difference between 30% and 55% MFU. Here is how the overlap actually works and how you confirm it on a trace.

MoE buys you more parameters for the same FLOPs, but the gating network, the load imbalance, and the all-to-all shuffle introduce failure modes dense models never see. Here is the training playbook.

RLHF with PPO is not one model training, it is four models in the loop at once, three of them on the GPU during every step. The memory math and the generation bottleneck are what trip people up.

On a thousand GPUs something fails every few hours. The question is not whether you lose a node, but how many GPU-hours you lose when you do. Checkpoint frequency, sharded writes, and fast restart decide that.

When all-reduce is the bottleneck, the cause is almost always a misconfigured path: traffic on the wrong link, a downed NIC, or a topology NCCL never discovered. Here is the systematic way to find the lost bandwidth.

Data, tensor, and pipeline parallelism all leave one sequence's activations on one device, so 200k+ token training hits a wall none of them can fix. The fourth axis shards the sequence itself, and the interview lives in the communication math.

Federated learning keeps data on-device, but it costs you accuracy, debuggability, and engineering complexity. The signal is naming when those costs are justified and when a simpler centralized pipeline with DP wins. Here is the answer.