prefix caching

A shared system prompt gets re-prefilled on every request unless the server remembers it. Prefix caching skips that work, but only when the blocks line up exactly. Here is the hashing and eviction machinery underneath.

Round-robin routing wastes a feature that doubles throughput: cache locality. Designing the router means choosing a replica by KV state, not just least-loaded. Here is the two-layer design.

GPU memory runs out long before you run out of conversations to cache. Offloading KV to CPU or NVMe buys capacity, but the interconnect can become the new bottleneck. Here is the bandwidth math.