fsdp

The interviewer wants you to know why a model that fits on no single GPU still trains, and to do the optimizer-state memory math that motivates sharding. Here is the parallelism taxonomy and the 16-bytes-per-parameter calculation.

FSDP and ZeRO let you train models too big for one GPU without splitting the compute. The signal is the ZeRO stages (what gets sharded at each level) and the communication you pay for it. Here is the answer.

ZeRO and FSDP both shard training state across data-parallel ranks, but the magic is in when each shard is gathered and freed. Get the memory accounting and the all-gather timing and you can size any run.