96How do you serve a Mixture-of-Experts model efficiently, and what makes expert parallelism hard?▼expertDeepSeekMistralNVIDIA2 replies◆ premiumMoE saves compute but is awkward to serve: experts must be sharded, tokens routed across devices, and batches balanced. The signal is the all-to-all communication and the load-imbalance problem, not the training story.Open full answer →
59What makes Mixture-of-Experts models hard to train, and how do you handle routing, load balance, and all-to-all?▼expertGoogle DeepMindMistralMeta1 replies◆ premiumMoE buys you more parameters for the same FLOPs, but the gating network, the load imbalance, and the all-to-all shuffle introduce failure modes dense models never see. Here is the training playbook.Open full answer →
