The LLM Gateway

TL;DR: An LLM gateway is a single proxy between your application and one or more model providers, where the cross-cutting concerns live: routing and fallback across models/providers, caching, rate limiting and quotas, authentication, cost tracking and budgets, observability (logging/tracing), and guardrails. Instead of every service calling providers directly with its own ad-hoc handling, you centralize it once. It also abstracts providers behind a common interface, reducing vendor lock-in. It is the operational backbone of a production LLM platform.

Why centralize

If every part of your app calls model providers directly, you duplicate (and inconsistently implement) the same concerns everywhere: retries, caching, auth, cost logging, rate limits, safety checks. The day a provider has an outage, you discover six services each handled fallback differently and two of them not at all. A gateway puts these in one layer that all traffic flows through, so policies are consistent, observable, and changeable in one place. Whether you build on an OpenAI-compatible proxy like LiteLLM, a managed product (Portkey, Cloudflare AI Gateway, Kong AI Gateway), or roll your own, the shape is the same: one control plane, many providers behind it.

What it handles

• Routing and fallback. Send each request to the right model (cheap vs frontier, see small vs large models) and fail over to another provider/model on a 429 or 5xx. Wrap each upstream in a circuit breaker so one provider's brownout does not stall every request behind timeouts. This is also the main defense against vendor lock-in: swap providers behind one interface.
• Caching. Exact-match and semantic caching to serve repeated or near-identical requests without paying the model again.
• Rate limiting, quotas, and auth. Per-tenant and per-user limits and access control (see rate limiting) so one noisy tenant cannot exhaust a shared provider quota.
• Cost tracking and budgets. Log tokens and cost per caller for attribution and alerts (see cost optimization), and hard-stop a team that blows its budget.
• Observability. Centralized logging, tracing, and metrics for every call (latency, tokens, errors, cache-hit rate), the basis for debugging and monitoring.
• Guardrails. Input and output safety checks applied uniformly (see guardrails) instead of reimplemented per service.

A worked example

Say you run a support-summarization feature: 5 million calls/month, where about 30% of prompts repeat near-verbatim (canned templates plus the same FAQ documents). Without a gateway, every call hits a frontier model at, very roughly, a few dollars per million input tokens. Add the gateway and three policies kick in: semantic cache absorbs the 30% repeats (those calls now cost a vector lookup, not a model call), a router sends the short, easy prompts to a small model at roughly an order of magnitude lower price, and a per-tenant budget caps runaway loops. The cache and routing together can cut the model bill by half or more, and the cap turns a runaway agent from a five-figure surprise into an alert. The gateway adds a few milliseconds of proxy overhead per call, trivial against the hundreds of milliseconds to seconds of model latency it sits in front of.

Why interviewers probe this

A production LLM platform has to handle all of these, and a candidate who would scatter them across services misses the architecture. A strong answer proposes a gateway as the single control plane for routing/fallback, caching, rate limiting, auth, cost, and guardrails, names a concrete tool (LiteLLM, Portkey), and reasons about the failure mode the gateway exists to prevent (a provider outage, a runaway budget, inconsistent safety). The reserved follow-up is usually "where does the gateway itself become the bottleneck or single point of failure?": answer that you run it stateless and horizontally scaled, keep cache and rate-limit state in a shared store like Redis, and degrade open or closed deliberately per route.

Common misconceptions

• "Just call the provider API directly." That scatters retries, caching, auth, cost, and safety across services and guarantees they drift apart; a gateway centralizes them.
• "A gateway is only for cost." Cost is one concern; it also handles routing/fallback, caching, rate limits, auth, observability, and guardrails.
• "It locks you into one provider." The opposite: abstracting providers behind one interface reduces lock-in and enables fallback.
• "It adds too much latency." Proxy overhead is a few milliseconds against multi-hundred-millisecond model calls, and caching/routing net far more than they cost.

Key takeaways

• An LLM gateway is a single proxy where cross-cutting concerns live: routing/fallback, caching, rate limiting, auth, cost, observability, guardrails.
• Centralizing makes policies consistent, observable, and changeable in one place, and prevents the per-service drift that bites during an outage.
• It abstracts providers behind one interface, reducing vendor lock-in and enabling failover with circuit breakers.
• Run it stateless and horizontally scaled with shared cache/rate-limit state so the control plane is not itself a single point of failure.