rate limiting

A favorite practical screen at the labs, because it tests concurrency, time handling, and judgment in 20 lines. The trap is the background thread that wastes CPU. Here is the lazy-refill version interviewers want, plus the follow-ups.

The most job-shaped coding screen in Applied AI: fan out N LLM calls without melting the rate limit or losing the batch to one bad request. The signal is in the retry policy, not the async syntax. Here is the version that passes.

Give every team an LLM API key and your bill becomes a mystery with a six-figure surprise at month end. Attributing, limiting, and optimizing spend is a platform feature. Here is how to build it.

Every API platform needs one. Interviewers reward the candidate who picks the right algorithm (token bucket vs sliding window) for the burst behavior, then solves the genuinely hard part: enforcing one shared limit across many servers without a per-request round trip to a central store.

LLM endpoints are uniquely exposed: each call can cost real money and burn scarce GPU capacity, and abuse is both a security and a financial problem. The signal is layered controls on tokens and cost, not just requests. Here is the defense.

An attacker doesn't need to breach your LLM to hurt you, just make it do expensive work. A handful of crafted prompts can pin your GPUs and run up the bill. The defense is not a bigger rate limit.