mixture of experts

Both are conditional compute, but one scales parameters and the other scales depth. The signal is knowing which axis each routes along and why that changes the FLOP story. Here is the sharp version.

MoE buys you more parameters for the same FLOPs, but the gating network, the load imbalance, and the all-to-all shuffle introduce failure modes dense models never see. Here is the training playbook.