AppliedAIPrep logoAppliedAI/Prep
← ML Infrastructure & GPUs / 56
hardMicrosoftMetaNVIDIA

Walk through the ZeRO stages and FSDP internals. Where does the memory actually go and when is it gathered?

ZeRO and FSDP both shard training state across data-parallel ranks, but the magic is in when each shard is gathered and freed. Get the memory accounting and the all-gather timing and you can size any run.

Updated Aug 2026 · Grounded in real Applied AI Engineer interview loops and written to a senior-engineer editorial bar.

ZeRO and FSDP both shard training state across data-parallel ranks, but the magic is in when each shard is gathered and freed. Get the memory accounting and the all-gather timing and you can size any run.

Unlock the other 750 answers · ₹2,000 / $25Your progress and mastery stay saved · 6 months · one payment · no auto-renew
UP NEXT ON YOUR JOURNEY
Next in this trackBeyond basic gradient checkpointing, how do you choose selective activation recomputation to maximize MFU?Next in this trackHow do you overlap communication with computation in distributed training, and how do you verify it works?Next in this trackWhat makes Mixture-of-Experts models hard to train, and how do you handle routing, load balance, and all-to-all?
DISCUSSION · 0

No comments yet — be the first to share your approach.