05Design a large-scale training pipeline that resumes cleanly after a node failure.▼hardNVIDIAOpenAIGoogle2 repliesunlockedAt thousand-GPU scale, hardware failure is the norm, not the exception, and a run that cannot resume wastes weeks. The signal is checkpointing strategy, deterministic resume, and minimizing lost work. Here is the fault-tolerant design.Open full answer →
39How do you cut training cost with spot/preemptible GPUs without losing days of work to a preemption?▼hardAWSGoogleDatabricks1 replies◆ premiumSpot GPUs are often 60-90% cheaper, and they vanish with two minutes' warning. The savings are only real if a preemption costs you minutes, not the run. Here is how to make that true.Open full answer →
61Design fault-tolerant checkpointing for a 1000-GPU training run. How do you minimize lost work on a failure?▼hardMetaNVIDIAMicrosoft2 replies◆ premiumOn a thousand GPUs something fails every few hours. The question is not whether you lose a node, but how many GPU-hours you lose when you do. Checkpoint frequency, sharded writes, and fast restart decide that.Open full answer →
