Rate Limiting, Retries, and Backoff

TL;DR: LLM apps call rate-limited, latency-heavy, occasionally-failing providers, so resilience is part of the design. Rate limiting (commonly a token bucket) protects your service and enforces per-tenant quotas. Retries with exponential backoff and jitter handle transient failures (a 429 or timeout) without hammering a struggling dependency. Circuit breakers stop sending traffic to a failing service so it can recover. Naive fixed-interval retries are dangerous, they synchronize and amplify load, turning a blip into an outage.

Why resilience is not optional here

LLM/provider calls are slow, expensive, and flaky: you hit rate limits (429s), timeouts, and provider outages routinely. A system that assumes calls always succeed quickly will fail badly. So you design for failure with three classic patterns.

Rate limiting (token bucket)

Rate limiting controls how many requests are allowed over time, protecting your own service from overload and enforcing per-tenant/user quotas (so one customer cannot starve others). The token bucket is the standard: tokens refill at a fixed rate, each request consumes one, and requests are rejected or queued when the bucket is empty, allowing bursts up to the bucket size while bounding the sustained rate.

Retries, backoff, and jitter

When a call fails transiently, retrying is reasonable, but how you retry matters:

• Exponential backoff: wait longer after each failure (1s, 2s, 4s...) so you do not hammer a struggling dependency.
• Jitter: add randomness to the wait so many clients do not retry in lockstep (a "thundering herd" that re-overloads the service). Backoff with jitter is the safe default.
• Bounded retries: cap the attempts and only retry idempotent/transient failures; do not retry a 400 (bad request) forever.

Circuit breakers

If a dependency is consistently failing, retrying at all just piles on load and ties up your resources waiting. A circuit breaker trips after a failure threshold and stops sending requests for a cooldown, then tentatively allows a few through to test recovery. This lets the failing service recover and fails fast for your users instead of hanging. Pair with graceful fallbacks (a cached or simpler response). These controls naturally live in the LLM gateway.

Why interviewers probe this

LLM calls are flaky enough that resilience is a real design concern, and naive retry logic is a common, dangerous mistake (synchronized retries amplify an incident). A strong answer covers rate limiting (token bucket, per-tenant quotas), retries with exponential backoff and jitter (not fixed intervals), and circuit breakers to stop hammering a failing dependency, ideally noting these belong in the gateway. That is production-grade thinking.

Common misconceptions

• "Just retry on failure." Fixed-interval retries synchronize and amplify load; use exponential backoff with jitter and bounded attempts.
• "Retry everything." Only retry transient/idempotent failures; retrying a bad request forever is pointless.
• "Rate limiting is only about cost." It also protects your service and enforces fair per-tenant quotas.
• "Keep retrying a down dependency." A circuit breaker should stop calls so it can recover; fail fast with a fallback.

Key takeaways

• LLM/provider calls are slow and flaky, so rate limiting, retries, and circuit breakers are part of the design.
• Token-bucket rate limiting protects your service and enforces per-tenant quotas while allowing bursts.
• Retry transient failures with exponential backoff plus jitter and bounded attempts, never fixed-interval retries.
• Circuit breakers stop traffic to a consistently-failing dependency so it can recover; pair with fallbacks.