latency

Token count is latency and dollars, not just style. 'Be concise' barely works, and max_tokens just truncates mid-sentence. Here is how to actually shape length without amputating answers.

Generation has to stop somewhere, and the wrong stop rule either truncates answers or burns tokens on trailing garbage. Here is how the EOS token, stop strings, and max-token caps actually interact.

Speculative decoding speeds up generation without changing the output distribution. The signal is knowing how draft-model, self-drafting (Medusa/EAGLE), and lookahead approaches differ in where the guesses come from.

Everyone budgets image tokens. Almost nobody budgets image latency. Images are a prefill problem, prompt caching stops paying for itself, and the cheapest answer is often not to call a VLM at all.

RAG and agents get expensive and slow fast: retrieval plus reranking plus big-model calls, multiplied across agent steps. The signal is naming the dominant cost first, then the levers that actually move it. Here is the playbook.

Voice is a major modality and a common applied-AI surface. The signal is the audio-to-text pipeline, why Whisper is resilient, and the cumulative latency budget that makes or breaks a real-time voice agent.

Accuracy ties are common, and accuracy is rarely the deciding factor anyway. The real tiebreakers are the ones a junior candidate forgets to mention. Here is the checklist a staff engineer runs before shipping either one.

Real-time transcription at scale tests streaming ASR, latency budgets, and GPU fleet management under heavy concurrency. The signal is streaming chunked inference with partial results, not batch transcription. Here is the design.

A voice assistant chains STT, an LLM, and TTS under a brutal latency budget where every stage adds delay. The signal is streaming and pipelining the stages plus turn-taking, not three blocking calls. Here is the design.

Every keystroke needs ranked suggestions back in tens of milliseconds, at search-engine QPS. The interesting part is the data structure and what you precompute, not the query path. Here is the design that survives the latency budget.

Predict, value, and bid on a single impression in about 10ms, billions of times a day. The signal is the pCTR-to-bid pipeline under a brutal latency budget, plus why calibration and budget pacing decide whether you make or lose money.

Translation is seq2seq generation served across hundreds of language pairs under tight latency and quality budgets. The signal is dodging the quadratic pair explosion, rescuing low-resource languages, and serving it cheaply. Here is the design.

Detecting and boxing objects at scale comes down to one driving tradeoff plus a handful of CV specifics most candidates fumble: the detector family, NMS, focal loss, and why mAP not accuracy is the metric. Here is the design.

Exact-match caching barely helps when no two prompts are identical. Semantic caching reuses answers for similar queries, and its whole risk is returning a near-match that's subtly wrong. Here is how to build it safely.

Inline completion lives or dies on tail latency: a suggestion that arrives after the developer keeps typing is useless. Learn how to build the context window, hide model latency behind speculation and caching, and keep a tight feedback loop on acceptance rate.

Moderating billions of items a day with an LLM on every one is too slow and too expensive. Learn the tiered funnel (cheap classifier then LLM then human), how to tune thresholds for precision versus recall, and how to keep up with adversaries.

A writing assistant has to feel instant while a cursor moves through a live document. Learn how to split fast deterministic checks from LLM rewrites, stream suggestions, and keep edits stable so the text does not flicker as the user types.

When a page loads, an ad slot is auctioned to dozens of bidders and a winner is chosen before the page finishes rendering, all in under 100ms. The design covers the auction mechanism, the brutal latency budget, budget pacing, and fraud. Here is how an exchange runs millions of auctions per second.

The trap is jumping straight to 'add more GPUs.' The signal is profiling first, then applying the cheap, no-retrain levers in the right order. Here is the diagnosis-then-optimize playbook for p99.

Caching is one of the biggest LLM cost levers, but 'cache the response' is naive for a non-deterministic system. The signal is distinguishing prompt (prefix) caching from semantic caching and knowing when each is safe.

Routing each request to the right model is one of the biggest LLM cost/latency levers in production. The signal is matching query difficulty to model capability and knowing when the cascade pattern beats a classifier.

A fraud model fed a feature an hour behind is half-blind, but recomputing everything in real time burns money you don't need to spend. Freshness is a per-feature decision on a real cost curve. Here is how to manage it.

Decode is sequential and memory-bound, so latency tricks matter. The signal is explaining speculative decoding's draft-and-verify mechanism (and why it stays exact) plus the other levers and when each applies. Here is the latency toolkit.

A freshly started replica is slow on its first requests while it loads weights and compiles kernels, and that clashes with reactive autoscaling. The signal is naming the cold-start sources and the warmup and provisioning fixes.

Users feel the p99, not the median, and the tail is where serving systems quietly fail. The causes are queuing and batching effects, not a slow model. Here is how to flatten it.

Launching an LLM service on a vibes-based GPU count is how you get paged on day one. Capacity planning for LLMs differs from web services because tokens, not requests, are the unit. Here is how to size it.

Three batching strategies, three very different latency profiles. Picking wrong leaves throughput or tail latency on the floor. Here is what each one costs and when to use it.

Forcing valid JSON sounds free, but the mask computation can stall every decode step. The fix is precompiling the grammar into a fast automaton. Here is where the cost hides and how to remove it.