decode

LLM inference has two phases with opposite resource profiles, and co-locating them makes a long prompt stall everyone else's tokens. The signal is knowing why prefill and decode fight, and what separating them costs. Here is the answer.

A single long prompt can freeze every other user's token stream for hundreds of milliseconds. Chunked prefill slices that prompt so decode keeps flowing. Here is the mechanism and the knob that controls it.