sharding

When you shard a vector index or route requests across model replicas, naive 'hash mod N' reshuffles everything when N changes. Consistent hashing moves only a small fraction. Here is the implementation.

A from-scratch test of distributed input pipelines. The signal is partitioning data across workers with no overlap and no gaps, epoch-consistent shuffling with a shared seed, and handling the uneven-last-batch problem. Here is the implementation.

Showing the top 10 is easy. Showing a player their global rank among 50M others, updated live, is the real challenge. The answer hinges on the right data structure, sharding, and approximate ranks for the long tail. Here is how leaderboards scale.

FSDP and ZeRO let you train models too big for one GPU without splitting the compute. The signal is the ZeRO stages (what gets sharded at each level) and the communication you pay for it. Here is the answer.

When a model is too big for one GPU you split the model itself, not just the data. The signal is distinguishing tensor parallelism (split within a layer) from pipeline parallelism (split across layers) and matching each to the interconnect. Here is the answer.