batching

Temperature 0 does not mean deterministic, and the reason is in the GPU kernels, not the sampler. The signal is naming the batch-invariance problem and knowing which fixes are real versus placebo.

Data loading is where training pipelines silently bottleneck. The signal is a correct shuffle-then-batch iterator plus knowing why prefetching and parallel loading keep the GPU fed.

Scarce GPUs, dozens of models, every team wanting low latency at low cost. The signal is whether you can turn that into one governed serving fleet: continuous batching, KV cache, per-tenant quotas, and cost you can actually attribute.

Summarizing a 200-page contract is not one LLM call: it is chunking, hierarchical reduction, and a faithfulness check so you never invent facts. Learn the map-reduce pattern, when long-context beats it, and how to evaluate summaries at scale.

The trap is jumping straight to 'add more GPUs.' The signal is profiling first, then applying the cheap, no-retrain levers in the right order. Here is the diagnosis-then-optimize playbook for p99.

Decode is sequential and memory-bound, so latency tricks matter. The signal is explaining speculative decoding's draft-and-verify mechanism (and why it stays exact) plus the other levers and when each applies. Here is the latency toolkit.

Knowing the algorithms is half the job. The other half is the serving stack that actually delivers throughput inside a latency budget. The signal is matching the server to the workload and naming the four knobs that move the needle.

Under load, a shared inference service must decide whose request runs now. The signal is queuing with priorities, backpressure, and fairness wired into batching, not first-come-first-served until the service falls over.

You rarely write a serving stack from scratch. The signal is knowing what frameworks hand you (batching, multi-model, GPU scheduling) and why LLM-specific servers exist at all when general ones already batch.

The counterintuitive truth of LLM serving: token generation is limited by how fast you can read weights from memory, not by math. Once you see that, the whole optimization menu falls out of one number.

Users feel the p99, not the median, and the tail is where serving systems quietly fail. The causes are queuing and batching effects, not a slow model. Here is how to flatten it.

Three batching strategies, three very different latency profiles. Picking wrong leaves throughput or tail latency on the floor. Here is what each one costs and when to use it.