TL;DR: A gateway is a unified API in front of many model providers that centralizes the concerns every LLM caller needs: routing (pick a model by cost, latency, or capability), caching (exact and semantic), fallback and retry across providers for reliability, rate limiting and cost governance per tenant, guardrails (PII, injection, output checks), and observability (logging, latency, token and cost metrics, tracing). It decouples application code from any single provider and enforces policy in one place.
How to approach it
Frame the motivation first: without a gateway, every service reimplements retries, key management, limits, logging, and provider quirks, and switching providers means touching all of them. The gateway centralizes these. Lay out the request path, then the cross-cutting features, and emphasize the reliability and governance angles that make it more than a proxy. The interviewer is checking whether you see the org-wide policy surface, not just the HTTP hop.
A strong answer
The request walks an ordered path; each stage is policy applied in one place.
The features a gateway centralizes:
- Unified API and provider abstraction. One interface (OpenAI-compatible, say) over many backends: OpenAI, Anthropic, open models on your own GPUs. App code does not change when you switch or add providers, and you can route the same call to different models.
- Routing. Choose the model per request by policy: cheapest model that meets the quality bar, fastest for latency-critical paths, a specific capability (vision, long context), tenant tier, or an A/B or canary split for model rollouts.
- Caching. Exact-match cache on normalized request keys, and optionally a semantic cache (embed the query, serve a cached answer for near-duplicates) since a meaningful share of traffic repeats. Big cost and latency win.
- Reliability: retries and fallback. Time out slow calls, retry transient failures with backoff, and fall back to an alternate provider or model when one is down or rate-limited, so a single provider outage does not take you down. A circuit breaker per provider stops you from hammering a dead backend.
- Rate limiting and cost governance. Per-tenant and per-key limits on tokens and spend, hard caps, and quotas, enforced centrally so a runaway loop in one app cannot drain the budget.
- Guardrails. Input PII redaction and injection checks, output filtering and validation, applied uniformly so every caller inherits them.
- Observability. Centralized logging of requests and responses (with PII handling), plus metrics for latency, error rate, tokens, and cost per tenant and per model, plus tracing. This is where you actually see and control LLM spend and quality.
The defensible framing: the gateway exists to centralize reliability, cost governance, safety, and observability so application teams do not each reinvent them, and so the org can switch models and enforce policy in one place.
Key takeaways
- A gateway is a policy chokepoint, not a proxy: routing, caching, fallback, limits, guardrails, and metrics all live in one place.
- Per-provider circuit breakers plus cross-provider fallback turn a single outage into a quality dip instead of an outage.
- Semantic caching pays off, but scope keys by tenant and context so you never serve one user's answer to another.
- Centralized token and cost metrics per tenant are the only way LLM spend stays governable as usage spreads.
What interviewers probe next
- "Semantic cache risks?" A near-duplicate query may need a different answer (different context or user), so scope cache keys carefully by tenant and context and set similarity thresholds conservatively.
- "How does fallback preserve quality?" Define equivalence classes of models so a fallback meets the same capability bar, and log when fallback fires so quality regressions are visible.
- "Streaming through a gateway?" Pass through token streams (SSE) while still logging usage at completion; do not buffer the whole response and kill time-to-first-token.
- "Where do guardrails belong, gateway or app?" Centralize baseline guardrails in the gateway for uniform enforcement; apps add domain-specific ones on top.
Common mistakes
- Treating it as a dumb proxy and missing the reliability (fallback), cost-governance, and observability value.
- No fallback, so one provider's outage or rate limit takes the whole product down.
- Caching without tenant or context scoping, leaking or mis-serving answers.
- No centralized token and cost metrics, so LLM spend is invisible and ungoverned.