19What is gradient (activation) checkpointing, and what does it trade off?▼mediumNVIDIAOpenAIGoogle1 replies○ sign inActivations, not just weights, can dominate training memory, and gradient checkpointing is the standard fix. The signal is the precise trade: recompute activations in the backward pass instead of storing them. Here is the answer.Open full answer →
57Beyond basic gradient checkpointing, how do you choose selective activation recomputation to maximize MFU?▼hardNVIDIAGoogleMeta2 replies◆ premiumFull activation checkpointing saves memory but burns a flat 30% extra compute. Selective recomputation recovers most of that by only recomputing the cheap, memory-heavy operations. Here is how to pick what to recompute.Open full answer →
