LLM Cost Optimization

TL;DR: LLM cost is driven by tokens (input + output) and the number of model calls, so the levers are: route easy queries to cheaper/smaller models, cache repeated and near-duplicate requests, trim context (fewer, better-retrieved chunks rather than giant prompts), use cheaper retrieval/reranking, and for agents cut unnecessary steps and parallelize. The discipline is to measure cost per request and attack the dominant contributor, not guess. Most teams overspend simply by sending every request to the largest model.

The cost model

Two things dominate the bill:

• Tokens per call (input context plus generated output), priced per token. Output tokens are typically 3 to 5x the price of input tokens, so verbose generation hurts more than a long prompt.
• Number of model calls (an agent makes many; reranking, query rewriting, and multi-step add calls).

Everything else (retrieval, embeddings) is usually smaller. So optimization targets tokens and calls.

The levers, by impact

• Model routing (often the biggest). Send the bulk of traffic to a cheaper/smaller model and reserve frontier/reasoning models for hard queries (see small vs large models). Since most traffic is easy, this is a large, direct win. Frontier vs small-model pricing differs by roughly 10 to 30x, so the routing split dominates everything else.
• Caching. Semantic and prefix caching serve repeated/similar requests for near-zero cost, huge on repetitive workloads. Provider prompt caching also discounts a stable prefix (system prompt, tool schemas, few-shot block) by up to ~90% on cache hits, so put the constant material first and the variable user turn last.
• Trim context. Better retrieval/reranking lets you pass fewer chunks; avoid stuffing long context you mostly ignore (see retrieval vs long context). Fewer input tokens cut cost and improve quality.
• Cheaper retrieval/reranking, quantized models, smaller max-output, and capping reasoning tokens on models that think before answering.
• Agents: reduce tool calls/iterations, prefer a fixed workflow when the path is known, and parallelize independent calls.

Worked example: where the money actually goes

A support assistant handles 1M requests/month. Naive design: every request hits a frontier model with 8K input tokens (full doc dump) and 500 output tokens. Take rough rates of $5 / 1M input and $15 / 1M output for the frontier model, and $0.15 / $0.60 for a small model.

Same product, roughly 20x cheaper, and the trimming usually raises answer quality because the model is not drowning in irrelevant chunks. Note the order: routing did the heavy lifting, context trimming was second, caching was the polish. Optimizing caching first would have been rearranging deck chairs.

Measure first

The cardinal rule: measure cost per request (and per feature/team/customer) before optimizing, then attack the dominant contributor, usually token volume or call count. Optimizing a small contributor wastes effort. The LLM gateway is where you instrument cost and apply routing/caching uniformly. Tie cost into the build-vs-buy and self-host-vs-API decisions.

Why interviewers probe this

Cost is a primary production constraint, and "would you send everything to the biggest model?" is a quick maturity check. A strong answer names the cost model (tokens and calls, output priced higher), the levers in impact order (routing, caching, context trimming, fewer agent steps), and the discipline of measuring cost per request before optimizing. That is exactly the lever applied-AI engineers pull to make a feature economically viable.

Common misconceptions

• "Use the best model for everything." Routing most traffic to a cheaper model is usually the biggest cost win.
• "Input and output tokens cost the same." Output is typically 3 to 5x pricier, so cap max-output and rein in verbose generation.
• "Cost is fixed by the provider." You control it via routing, caching, and context size.
• "Optimize wherever." Measure cost per request and attack the dominant contributor (tokens/calls), not a minor one.
• "Bigger context is free with large windows." It is priced per token and slower; fewer, better chunks usually win.

Key takeaways

• LLM cost is dominated by tokens (output pricier than input) and the number of model calls.
• The biggest levers are routing easy queries to cheaper models and caching repeated/similar requests.
• Trim context (fewer, better chunks) and cut unnecessary agent steps.
• Measure cost per request first and attack the dominant contributor; centralize controls in the gateway.