TL;DR: A Mixture-of-Experts layer replaces one big feed-forward network with many expert FFNs plus a router that sends each token to only a few experts (top-k). Total parameters are large but compute per token is small (sparse activation): you get the quality of a big model at the FLOPs of a small one. The catch: you must hold all experts in memory, routing can become unbalanced, and serving is more complex.
How to approach it. Lead with the core idea (decouple parameter count from per-token compute via sparse routing), then the mechanism (router plus top-k experts), then be honest about the costs the FLOP win obscures: memory, load balancing, and serving complexity. That balance separates a real answer from a hype answer.
A strong answer. The idea. In a dense transformer every token passes through the same feed-forward network, so compute scales with total parameters. MoE replaces that FFN with N expert FFNs and a small router (gating network) that, per token, picks the top-k experts (often k=2) to actually run. A model can hold 8 or 64 experts' worth of parameters but activate only 2 per token. Total parameters are large, active parameters per token are small. That scales capacity (which helps quality) without scaling per-token FLOPs, which is why Mixtral and DeepSeek use it.
Routing. The gating network produces a distribution over experts for each token and selects top-k; their outputs are combined, weighted by the gate. Routing is learned jointly with the rest of the network. Because the choice is discrete (which experts fire), it needs care: a naive router collapses to always using a few experts, so training adds a load-balancing loss to push utilization to spread evenly.
The tradeoffs the FLOP win hides:
- Memory. You must keep all experts resident (or shard them across devices) even though only k fire per token. MoE saves compute, not memory; an 8x7B-class model (about 47B total parameters) needs the memory of all experts, which is why MoE models are memory-hungry to host.
- Load balancing. Skewed routing overloads some experts and idles others, wasting capacity and hurting throughput. The auxiliary balancing loss and capacity factors manage this, imperfectly.
- Serving complexity and latency variance. Expert parallelism means tokens in a batch route to different devices, adding all-to-all communication and making batching and latency less uniform than a dense model.
- Training instability. Routing is finicky; MoE models historically needed tricks to train stably.
The defensible framing: MoE buys quality-per-FLOP and per-token cost, at the price of memory footprint, routing complexity, and harder serving. Reach for it when you want big-model quality at small-model inference compute and can afford the memory and infra.
Key takeaways
- MoE decouples total parameter count from per-token compute via top-k sparse routing.
- It saves FLOPs and latency, not memory: every expert must stay resident.
- The router and its load-balancing loss are what make MoE actually train and serve.
- Reason in active parameters for cost and total parameters for memory; conflating them is the classic error.
What interviewers probe next.
- "Does MoE reduce serving cost?" It reduces compute per token, not memory: you still host all experts, so memory and infra cost stay high. The win is FLOPs and latency for a given quality.
- "Why the load-balancing loss?" Without it the router collapses to a few favored experts, wasting the rest and creating hotspots; the aux loss spreads tokens across experts.
- "How is it parallelized?" Expert parallelism shards experts across devices; tokens route with all-to-all communication, a real overhead.
- "GShard/Switch Transformer vs newer MoE?" The lineage went from many-expert top-2 routing toward fine-grained experts and shared experts (DeepSeek), trading granularity against stability.
Common mistakes.
- Saying MoE makes the model cheaper to host; it cuts compute per token, not memory (all experts must be resident).
- Forgetting the router and load-balancing loss, the parts that make it work.
- Conflating total parameters with active parameters when reasoning about cost.
- Ignoring the serving and communication complexity that MoE adds versus a dense model.