routing
Applied AI interview questions tagged routing, across every topic.
3 questions · 2 unlocked for you
Concepts behind "routing"
The curriculum that explains the ideas these questions test.
Foundational
The LLM GatewayAn LLM gateway is a single proxy layer between your application and one or more model providers. It centralizes the cross-cutting concerns every LLM app needs: routing and fallback across models/providers, caching, rate limiting, authentication, cost tracking, observability, and guardrails. It also prevents vendor lock-in by abstracting providers behind one interface. Applied-AI interviews probe it because it is the backbone of a production LLM platform and the place most operational controls live.⚙️ System Design for AI in Production
Foundational
LLM Cost OptimizationLLM systems get expensive fast, and the cost model is mostly tokens and number of model calls. The levers, in rough order of impact: route easy queries to cheaper/smaller models, cache repeated and similar requests, trim context (fewer, better chunks), use cheaper retrieval/reranking, and for agents cut unnecessary steps. The discipline is measuring cost per request and attacking the dominant contributor. Applied-AI interviews probe it because cost is a primary production constraint and most teams overspend by defaulting to the biggest model on everything.⚙️ System Design for AI in Production
Core
Mixture-of-ExpertsA Mixture-of-Experts model replaces the dense feed-forward layer with many expert networks and a router that sends each token to only a few of them. This decouples total parameters (capacity) from per-token compute: the model can be huge while each token activates only a slice. The trade-offs are routing complexity, memory (all experts must be loaded), and load balancing. Applied-AI interviews probe it because most frontier models are MoE, and it explains how models get more capable without proportionally more inference cost.🧠 Foundations of LLMs & GenAISign in
Core
Small vs Large Models and RoutingBigger is not always better in production: small models are far cheaper and faster, and for many tasks they are good enough, especially when fine-tuned or given retrieval. The mature pattern is routing, send easy queries to a small/cheap model and reserve large or reasoning models for genuinely hard ones, often with a cascade that escalates on low confidence. Applied-AI interviews probe it because picking and routing models is where most of the cost and latency budget is won or lost.🧠 Foundations of LLMs & GenAISign in