TL;DR: Match the server to the workload. For LLM token generation, pick an LLM-specialized server (vLLM, TGI, TensorRT-LLM) that does continuous batching and PagedAttention, the two features that actually drive LLM throughput. For general or multi-framework models (vision, classical), Triton or TorchServe handle dynamic batching, multi-model, and GPU sharing. The recurring knobs are batching strategy, max concurrency, KV-cache memory, and quantization. Tune them to your latency SLO versus throughput target, then prove it under load.
How to approach it. Split the problem on the first sentence: LLM decode versus everything else. LLM generation is sequential and memory-bound, so it lives or dies on continuous batching plus paged KV. A vision or tabular model is a single forward pass that wants dynamic batching and packing. Name the representative server for each, then the four configuration levers tied to the throughput-versus-latency tradeoff. Ask one clarifying question: what is the p99 latency budget and the target QPS.
A strong answer. Why the workload split matters. An LLM generates tokens autoregressively, so a request-at-a-time or static-batch server wastes the GPU because sequences finish at different times. The wins come from continuous (in-flight) batching and PagedAttention (paged KV-cache storage), so you reach for an LLM-specialized server:
- vLLM: continuous batching plus PagedAttention, high throughput, OpenAI-compatible API, broad model support. The common default for self-hosted LLMs.
- TGI (Text Generation Inference): comparable capabilities, tight Hugging Face ecosystem integration.
- TensorRT-LLM / Triton: NVIDIA's optimized path (compiled kernels, FP8, in-flight batching) for peak performance on NVIDIA hardware.
For general models (vision, recommendation, classical ML, mixed frameworks), the bottleneck is a single forward pass across many models, so the priorities shift:
- Triton Inference Server: multi-framework (TensorRT, ONNX, PyTorch, TF), dynamic batching, concurrent model execution, GPU sharing, and ensembles. The strong general-purpose pick.
- TorchServe / ONNX Runtime / Ray Serve: PyTorch-native serving, or Python orchestration (Ray Serve) when you need custom pre/post-processing and composition.
| Workload | Server | Throughput lever |
|---|---|---|
| LLM token generation | vLLM, TGI, TensorRT-LLM | Continuous batching, PagedAttention |
| Vision / single-pass | Triton, TorchServe | Dynamic batching, model instances |
| Multi-model / mixed framework | Triton, Ray Serve | Concurrent instances, GPU sharing |
The four knobs that actually move throughput:
- Batching strategy. Dynamic batching groups concurrent requests (general models). Continuous batching admits and evicts sequences every decode step (LLMs). Set max batch and max wait so p99 stays inside budget.
- Concurrency / model instances. Run multiple model copies per GPU when memory allows, or shard across GPUs, sized to avoid queue buildup.
- KV-cache / memory. For LLMs, set the KV-cache memory fraction and max sequence length. PagedAttention cuts fragmentation so you pack more concurrent sequences.
- Quantization / precision. Serve INT8 or FP8 to fit more concurrency and lift token rate, after checking quality holds.
Add tensor or pipeline parallelism only when the model does not fit on one GPU. The defensible call: LLM-specialized server for generation, Triton or TorchServe for general and multi-model, then tune the four knobs to your specific SLO and validate with a load test, not a spreadsheet.
Key takeaways.
- The first decision is workload type: LLM decode (continuous batching, paged KV) versus single-pass (dynamic batching).
- vLLM or TGI for self-hosted LLMs, Triton for multi-framework and multi-model serving.
- Four knobs carry most of the gain: batching, concurrency, KV-cache memory, precision.
- Throughput numbers are only real after a load test that holds your p99 SLO.
What interviewers probe next.
- "Why not just put the model behind a Flask app?" No batching, idle GPU, no concurrency control. You get a small fraction of a real server's throughput.
- "Continuous versus dynamic batching?" Dynamic groups whole requests (single-pass models). Continuous admits and retires sequences mid-generation, which is what variable-length LLM output needs.
- "How do you set batch size and max wait?" Work backward from the SLO: bigger batches lift throughput but add queue wait, so cap max-wait where p99 still clears.
- "Multiple models on one GPU?" Triton runs concurrent instances and shares the GPU. Size by memory and watch for compute contention.
Common mistakes.
- Serving an LLM on a general server or bare web app with no continuous batching or paged KV, leaving most throughput on the table.
- Treating every model the same when LLM decode and a vision forward pass have different bottlenecks.
- Cranking batch size for throughput and blowing the latency SLO.
- Shipping a configured stack without a load test, so real p99 and capacity are guesses.