prefill

LLM inference has two phases with opposite resource profiles, and co-locating them makes a long prompt stall everyone else's tokens. The signal is knowing why prefill and decode fight, and what separating them costs. Here is the answer.

A single long prompt can freeze every other user's token stream for hundreds of milliseconds. Chunked prefill slices that prompt so decode keeps flowing. Here is the mechanism and the knob that controls it.

A shared system prompt gets re-prefilled on every request unless the server remembers it. Prefix caching skips that work, but only when the blocks line up exactly. Here is the hashing and eviction machinery underneath.