TL;DR: Rate-limit and quota on the dimensions that actually cost you (tokens and compute, not just request count), per user/key/tenant, with hard spend caps and alerts. Add abuse detection (anomalous usage, scraping, jailbreak attempts), authentication and per-key scoping, and graceful degradation under load. The distinctive part versus a normal API: cost is per token and capacity is scarce, so you govern consumption, not just traffic.
How to approach it. Call out what makes LLM endpoints different: each request consumes variable, expensive resources (tokens, GPU time), so a flat requests-per-minute limit is insufficient. Then layer the controls: identity, token/cost-based limiting, quotas and caps, abuse detection, and degradation.
A strong answer. What is different. A normal API request is cheap and roughly uniform; an LLM request can generate thousands of tokens, cost real money, and occupy scarce GPU capacity for seconds. A 10-token completion and a 10,000-token one count identically under requests-per-minute but differ 1000x in cost. So you must govern consumption (tokens, compute, dollars), not merely request count, or a few heavy users (or an attacker) can run up the bill and starve everyone else.
Layered controls:
- Authentication and scoping. Every call carries an authenticated key; scope keys to allowed models, endpoints, and limits. No anonymous access to expensive capacity.
- Rate limiting on the right dimensions. Limit requests per minute and tokens per minute (and concurrent requests) per key/user/tenant, using a token-bucket or sliding-window limiter, enforced server-side in a shared store (Redis with atomic Lua) for multi-node. Tokens-per-minute is the limit that maps to real cost and capacity.
- Quotas and hard spend caps. Daily and monthly token or dollar budgets per account, with a hard cap that blocks or downgrades when exceeded, plus alerts well before the ceiling. This prevents a bug or a compromised key from generating an unbounded bill.
- Abuse and anomaly detection. Flag spikes, scraping patterns, distributed-key abuse, and content-policy or jailbreak attempts; throttle or block offenders. Log usage per key for forensics.
- Cost-aware routing and degradation. Under load or for lower tiers, route to smaller/cheaper models, cap max output tokens, and shed or queue low-priority traffic so the platform degrades gracefully rather than melting down. Prioritize paying or critical tenants.
- Output and input guardrails. Block policy-violating content and prompt-injection attempts, which are also a form of abuse.
The flow of a single call:
The defensible framing: protect the two scarce resources (money and GPU capacity) with token/cost-based limits and hard caps, identify and isolate abusers, and degrade gracefully so one user cannot harm the rest.
Key takeaways.
- Govern consumption (tokens, compute, dollars), not request count; requests vary 1000x in cost.
- Hard per-key spend caps plus alerts are the only thing that bounds a leaked or buggy key.
- Enforce limits server-side in a shared counter (Redis atomics), never client-side or per-node.
- Degrade gracefully under contention: cheaper models, output caps, priority tiers, shed low-priority load.
What interviewers probe next.
- "Why not just requests-per-minute?" Requests vary enormously in cost (10 tokens vs 10,000); token/compute-based limits reflect actual spend and capacity, which is what you are protecting.
- "Distributed rate limiting across many servers?" Centralize counters in Redis with atomic operations (Lua script) so the limit holds globally, not per node.
- "How do you stop a runaway bill from a leaked key?" Hard spend caps per key, anomaly alerts, instant key revocation/rotation, and max-output-token limits per request.
- "Fairness under contention?" Per-tenant quotas and priority tiers so heavy users cannot starve others; queue or shed low-priority load.
Common mistakes.
- Limiting only request count, so token-heavy calls blow past cost/capacity controls.
- No hard spend cap, so a bug or compromised key generates an unbounded bill.
- Enforcing limits only in the client or per-node, leaving them bypassable.
- No graceful degradation, so a load spike takes down the whole service instead of shedding low-priority traffic.