llm serving

The canonical AI system-design question. The signal is covering both the model lifecycle and the serving stack under real scale, without rambling. Most candidates design only half and lose the points.

Inline completion lives or dies on tail latency: a suggestion that arrives after the developer keeps typing is useless. Learn how to build the context window, hide model latency behind speculation and caching, and keep a tight feedback loop on acceptance rate.

Personalizing an LLM is a retrieval and memory problem, not a per-user fine-tune. Learn how to assemble user context at request time, manage long-term memory without bloating the prompt, and respect privacy and the right to be forgotten.

A writing assistant has to feel instant while a cursor moves through a live document. Learn how to split fast deterministic checks from LLM rewrites, stream suggestions, and keep edits stable so the text does not flicker as the user types.

CPU-based autoscaling that works fine for a web tier quietly fails on GPU inference: wrong signal, and replicas that take minutes to warm. The interviewer wants the signals you actually scale on and how you hide the cold start.