System Design for AI in Production

This is two questions hiding as one: how would you have caught it before full rollout, and how do you debug it now. Answer both and you signal senior judgment. Here is the staged-rollout and root-cause playbook.

The modal ML system design round. The structure interviewers reward is the funnel: candidate generation then ranking then re-ranking, with the right model at each stage and an honest plan for cold start, freshness, and feedback loops. Here is that structure.

Extreme class imbalance plus a hard latency budget plus an adversary who adapts. The signal is handling imbalance honestly, choosing the operating point from costs, and designing for the feedback loop. Here is the end-to-end design.

Models fail silently; the question is whether you would know. The signal is monitoring the right layers (operational, data, prediction, outcome) and alerting on drift without drowning in false pages. Here is the system and the metrics that matter.

Multimodal search tests whether you understand shared embedding spaces and the retrieve-then-rank pattern under a real catalog's scale and freshness. The signal is CLIP-style joint embeddings plus hybrid retrieval and honest relevance evaluation. Here is the design.

Text-to-SQL is deceptively hard because correctness is binary and the failure mode is a confident wrong number. The signal is schema grounding, query validation, and a safety layer, not just 'prompt an LLM with the schema.' Here is the production design.

Moderation is a multi-stage classification problem with brutal tradeoffs: false negatives cause real harm, false positives censor legitimate users, and the adversary adapts. The signal is the tiered pipeline, per-severity precision/recall calibration, and human-in-the-loop. Here is the design.

As soon as a company uses LLMs in more than one place, it needs a gateway. The signal is the cross-cutting concerns (cost, reliability, observability, governance) that a gateway centralizes, not just 'proxy the API.' Here is the design.

Ads ranking is where calibrated probabilities meet hard latency and money. The signal is knowing that CTR must be calibrated (not just ranked), the feature and serving design, and the auction context. Here is the design that goes beyond 'train a classifier.'

Anomaly detection sounds easy until seasonality, cold start, and alert fatigue hit. The signal is modeling the expected baseline (including weekly and daily cycles), choosing unsupervised methods for scarce labels, and tuning to avoid drowning users in false alarms. Here is the design.

The canonical AI system-design question. The signal is covering both the model lifecycle and the serving stack under real scale, without rambling. Most candidates design only half and lose the points.

The modern agentic system-design question. The signal is the plan, search, read, synthesize, verify loop with per-claim citations, plus bounding cost and latency. The hard parts are not the writing.

Cross-session memory is what makes an assistant feel personal, and it is mostly a retrieval and state-management problem, not a bigger context window. The signal is the tiered architecture plus what to store, forget, and protect.

Customer support is the killer applied-AI use case, testing agents, RAG, tools, and the all-important human escalation. The signal is knowing when to resolve, when to act, and when to hand off, safely.

Scarce GPUs, dozens of models, every team wanting low latency at low cost. The signal is whether you can turn that into one governed serving fleet: continuous batching, KV cache, per-tenant quotas, and cost you can actually attribute.

AI code review lives or dies on precision: a few noisy comments and the team mutes the bot forever. The signal is grounding in the diff plus repo context, ruthless false-positive control, and earning developer trust one accepted comment at a time.

Summarize, draft, triage, all over the most sensitive PII a person owns. The signal is grounding in the actual thread, matching the user's voice, and a hard human-in-the-loop rule on anything that gets sent.

A GPU-heavy generative system where sampling steps dominate the bill and one bad image is a headline. The signal is the diffusion serving pipeline, the cost levers on per-image GPU time, and a mandatory two-sided safety layer.

Resume screening is high-scale and high-stakes: a legally sensitive, bias-prone decision about people. The signal is balancing throughput with fairness, human oversight, and explainability, not an LLM ranking resumes. Here is the design.

Real-time transcription at scale tests streaming ASR, latency budgets, and GPU fleet management under heavy concurrency. The signal is streaming chunked inference with partial results, not batch transcription. Here is the design.

Document extraction (IDP) is a huge enterprise use case with a hard correctness bar: a wrong extracted number is worse than none. The signal is the parse, extract, validate, human-review pipeline and confidence-based routing. Here is the design.

A voice assistant chains STT, an LLM, and TTS under a brutal latency budget where every stage adds delay. The signal is streaming and pipelining the stages plus turn-taking, not three blocking calls. Here is the design.

Healthcare is the highest-stakes AI domain: a wrong answer can harm a patient, and regulators are watching. The signal is designing decision support with grounding, human oversight, and guardrails, never an autonomous diagnoser. Here is the design.

Legal review is high-stakes and precision-critical: a missed clause or hallucinated citation has real consequences, and lawyers have been sanctioned for fabricated cites. The signal is grounding in the actual documents, citing exact passages, and lawyer-in-the-loop. Here is the design.

Dynamic pricing blends demand forecasting, optimization, and real-time serving, with fairness and trust constraints. The signal is the predict-then-optimize structure plus guardrails against perverse outcomes. Here is the design.

E-commerce search is where retrieval, ranking, and business goals collide, and it carries a hard exact-match requirement (brand, size, SKU). The signal is query understanding plus hybrid retrieval plus business-aware ranking. Here is the design.

A GPU-heavy generative-audio system with text/genre/lyric conditioning, minutes-long coherence, and copyright constraints. The signal is the generation pipeline plus the async, cost, and safety layer that makes it shippable.

Video generation is image generation plus the brutal constraint of temporal consistency, and it is the most GPU-intensive generative task there is. The signal is diffusion over spacetime plus a serving design that survives multi-minute jobs.

Code migration is a high-value agentic task with an unforgiving correctness bar: the migrated code must still work. The signal is grounding in the real repo, transforming in verifiable chunks, and gating on tests, never one giant LLM rewrite.

Notification systems fail by spamming users into muting them, and a mute is a permanently lost channel. The signal is treating it as a per-user send/hold/batch/suppress decision optimized for long-term trust, not for clicks or volume.

Two models chained (ASR then summarization), a long-context problem hiding inside, and a faithfulness bar where a fabricated action item has real consequences. The signal is the pipeline plus how you keep a 3-hour transcript honest at thousands a day.

On-device AI trades raw capability for privacy, offline use, and latency, under brutal memory and battery limits. The signal is the small-quantized-model stack plus a hybrid router that escalates the hard queries to the cloud without leaking private context.

The trap is replacing the classifier with an LLM. The real-time, imbalance, and adversarial constraints do not go away. The signal is a hybrid: a fast calibrated model scores inline, LLMs investigate the gray zone off the hot path.

How large recommenders and search pull candidates from millions of items in milliseconds. The signal is why the user and item towers are separate, how that unlocks precomputed embeddings plus an ANN index, and where ranking takes over.

Every keystroke needs ranked suggestions back in tens of milliseconds, at search-engine QPS. The interesting part is the data structure and what you precompute, not the query path. Here is the design that survives the latency budget.

ETA is spatiotemporal regression served in real time over conditions that change by the minute. The signal is feature freshness, segment or stage decomposition, and the asymmetric cost of being wrong. Here is the design.

PYMK is graph recommendation at billion-node scale. The interviewer is screening for one instinct: do you generate candidates from the social graph, or naively try to score all pairs? Here is the design that survives the follow-ups.

Feed ranking is multi-objective ML at massive scale with a brutal serving constraint. The interviewer wants to see whether you optimize a value model toward long-term satisfaction, or fall into the CTR trap that breeds clickbait. Here is the design.

The service behind every RAG stack and semantic search box. The signal is your ANN index choice and how you handle the three things that quietly break it: metadata filtering, live updates, and scale.

Predict, value, and bid on a single impression in about 10ms, billions of times a day. The signal is the pCTR-to-bid pipeline under a brutal latency budget, plus why calibration and budget pacing decide whether you make or lose money.

Image search is embedding retrieval at billion-image scale. The tell of a strong answer is knowing when a vision-only encoder suffices and when you need a CLIP-style shared space, plus how the ANN index actually serves the query. Here is the design.

Spam detection is adversarial, imbalanced classification under a low-latency bar. The interviewer is watching for whether you set the threshold by cost asymmetry and lean on signals spammers cannot fake. Here is the design.

Query understanding is the front end of search that turns a raw query into structured intent. Get it wrong and even a perfect index returns garbage. The signal is the pipeline and how you measure it. Here is the design.

Online models need features that are fresh and computed exactly the way training computed them. The classic failure is training-serving skew that silently degrades the model. The signal is the online/offline split with one shared definition. Here is the design.

New users and new items have zero interaction history, so collaborative filtering has nothing to work with. The candidates who pass treat it as a lifecycle, not a single trick. Here is how.

Rank each item by relevance alone and you ship ten near-identical results and a filter bubble. The signal interviewers want is a re-ranking stage that scores the list as a set, plus the metric that proves it works.

Some questions need precise multi-hop facts that vector retrieval can't chain. The signal is entity linking plus query translation over a graph (or GraphRAG), and knowing when the graph is worth its maintenance cost. Here is the design.

AV perception is safety-critical, real-time, multimodal ML: detect and track everything around the vehicle from multiple sensors. The signal is sensor fusion plus detection/tracking under a hard latency budget and a fail-safe bar. Here is the design.

Translation is seq2seq generation served across hundreds of language pairs under tight latency and quality budgets. The signal is dodging the quadratic pair explosion, rescuing low-resource languages, and serving it cheaply. Here is the design.

Recognizing walking, driving, or a fall from a sensor stream is windowed time-series classification, usually on a battery-bound device. The signal is windowing, temporal smoothing, and the on-device constraints most candidates skip. Here is the design.

Matrix factorization is the classic collaborative-filtering method and the conceptual root of modern embedding-based recsys. The signal is the latent-factor idea and how it leads to two-tower/neural models.

Recommending events (or jobs) breaks normal recsys in two places: items expire and items are local. The signal is how you handle perishability, geography, and the fact that every item is a cold-start item. Here is the design.

Tagging articles at scale is bread-and-butter ML, but the easy version fails three ways: it is multi-label, the taxonomy is hierarchical and keeps growing, and rare classes hide behind aggregate accuracy. The signal is the model choice plus how you handle all three.

A recommender trains on data its own past recommendations produced, so it learns to confirm its own beliefs. The signal is recognizing the loop, naming the biases it breeds, and knowing the exploration and debiasing fixes that break it.

Labeled data is the fuel for ML, and a labeling platform lives or dies on quality control. The signal is the workflow plus the quality math: consensus, gold honeypots, inter-annotator agreement, and active learning to spend the budget where it counts.

Detecting and boxing objects at scale comes down to one driving tradeoff plus a handful of CV specifics most candidates fumble: the detector family, NMS, focal loss, and why mAP not accuracy is the metric. Here is the design.

Two forces decide this design: documents longer than the context window, and summaries that must not lie. The strong answer pairs hierarchical map-reduce with claim-level grounding, then serves it cheaply at scale.

A high-AUC churn model that changes no behavior is worthless. The strong answer nails the label definition, picks classification vs survival deliberately, and is judged on retention uplift, not accuracy.

Sales has finite rep-hours, so the model's job is to point them at the leads worth a call. The signal interviewers want is value-weighted prioritization, careful conversion labels, and proof of revenue uplift over a holdout.

Forecasting one series is a textbook exercise; forecasting a million SKU-stores that must reconcile, with stockouts costing more than overstock, is the real interview. The strong answer goes global, hierarchical, and cost-aware.

A Devin-style agent that turns an issue into a merged PR is the hardest agent to make reliable, because every step can fail and errors compound. Here is the architecture and the loop that keeps it honest.

Exact-match caching barely helps when no two prompts are identical. Semantic caching reuses answers for similar queries, and its whole risk is returning a near-match that's subtly wrong. Here is how to build it safely.

GPUs are scarce and expensive, so multi-region HA for LLMs is not just web-app HA with bigger boxes. Capacity, routing, and failover all bend around the GPU constraint. Here is the design.

Batch scoring a billion items is a throughput-and-cost problem, the opposite of low-latency serving. Every choice that hurts latency helps you here. Here is how to design for dollars-per-million-documents.

Every team eval-ing prompts in ad-hoc notebooks is how an org ships regressions and argues about vibes. A shared eval platform makes quality measurable and comparable. Here is what it has to provide.

Prompts buried in source code mean every wording tweak is a full deploy, and nobody knows which prompt is live. Treating prompts as managed, versioned configuration is the fix. Here is the platform.

An agent that clicks around a real website is slow, brittle, and one wrong click from a costly mistake. The architecture is about perception, action, and guardrails around irreversible steps. Here is how to build it.

RLHF and evals are only as good as the preference data behind them, and that data is generated by humans whose quality varies wildly. The platform that produces trustworthy labels is itself a serious system. Here is its design.

'What was revenue by region last quarter?' sounds like text-to-SQL, but production analytics agents fail on ambiguity, wrong joins, and confidently wrong numbers. Here is the architecture that makes the answers trustworthy.

Give every team an LLM API key and your bill becomes a mystery with a six-figure surprise at month end. Attributing, limiting, and optimizing spend is a platform feature. Here is how to build it.

An AI support agent handling millions of chats will sometimes be wrong, rude, or unhelpful, and you cannot read them all. Monitoring quality at that scale is its own design problem with its own failure modes.

Classic distributed-systems tradeoffs show up in ML infra with an ML twist: stale features and eventually-consistent reads have model-accuracy consequences, not just correctness ones. Here is how to reason about it.

Anthropic's signature opener. The naive answer (one server pushes to N clients) is bottlenecked by the source uplink and scales linearly. The answer interviewers reward turns receivers into senders so the swarm's capacity grows with its size. Here is that reasoning.

The Dynamo question. Interviewers want three decisions made cleanly: how you partition keys so adding nodes does not reshuffle everything, how you replicate for durability, and where you land on the CAP spectrum. Here is how to reason about all three with concrete quorum math.

Every API platform needs one. Interviewers reward the candidate who picks the right algorithm (token bucket vs sliding window) for the burst behavior, then solves the genuinely hard part: enforcing one shared limit across many servers without a per-request round trip to a central store.

Looks simple until the follow-ups: which layer do you balance at, how do you avoid sending traffic to a dead server, and how is the load balancer itself not a single point of failure? Interviewers want the algorithm, the health-check loop, and the high-availability story.

A deceptively rich warm-up. The interesting decisions are how you generate short, collision-free, unguessable keys at scale, and how a read-heavy 100:1 workload shapes the storage and caching. Here is the clean design with capacity math.

The observability infrastructure question. Interviewers want you to separate the three telemetry types, choose storage that survives massive write volume, and design alerting that fires fast without drowning on-call in noise. Here is the pipeline with retention and cardinality math.

The distributed ordering question, the conceptual core of Kafka, replicated logs, and consensus systems. Interviewers want to see you confront the hard truth that physical clocks lie, then reach for the actual tools: a single sequencer, consensus, or logical clocks. Here is that progression.

The real-time messaging classic. Interviewers want the delivery path (persistent connections, not polling), the fan-out strategy that does not collapse on large groups, and an ordering and delivery-guarantee story. Here is that design with the fan-out tradeoff that separates seniors.

This is the question to design the scoreboard, not the player. A strong answer separates offline gates (AUC, calibration, NDCG) from the online verdict (A/B with revenue, user, and advertiser guardrails), and adds counterfactual replay so you can trust a model before it ever serves a real auction.

A news feed is not a generic recommender. The clock is a first-class signal: a story that mattered this morning is noise by tonight. The interviewer wants recency decay, an engagement-versus-quality value model, and an answer for filter bubbles, not just retrieval plus ranking.

Truth is not a label you can train on cheaply, and adversaries adapt the moment you ship. A strong answer fuses content, graph, and behavioral signals, puts humans in the loop where precision matters, and treats adversarial drift as a permanent operating condition, not a one-time training problem.

Detecting the language of a string sounds trivial until the input is three words, mixes two languages, or is half code. A strong answer picks the model by latency budget, handles short text and code-switching, and returns calibrated confidence instead of a brittle single guess.

Cleaning noise out of speech is a tradeoff between how good it sounds and how fast it runs. A strong answer picks spectrogram masking versus waveform models against a real-time budget, names PESQ and STOI as the metrics, and is honest about the artifacts each approach introduces.

Video search is image search with a time axis, and the time axis is the whole problem. A strong answer embeds frames, pools them into scene vectors, indexes with ANN at billion scale, and explains how a multimodal query (clip, text, or both) finds the right moment, not just the right video.

Inline completion lives or dies on tail latency: a suggestion that arrives after the developer keeps typing is useless. Learn how to build the context window, hide model latency behind speculation and caching, and keep a tight feedback loop on acceptance rate.

Enterprise search lives or dies on permissions, freshness, and connecting to thirty messy SaaS sources; retrieval quality is table stakes. Learn how to fan out across connectors, enforce per-user access at query time, and blend lexical with vector search for results people trust.

Summarizing a 200-page contract is not one LLM call: it is chunking, hierarchical reduction, and a faithfulness check so you never invent facts. Learn the map-reduce pattern, when long-context beats it, and how to evaluate summaries at scale.

Moderating billions of items a day with an LLM on every one is too slow and too expensive. Learn the tiered funnel (cheap classifier then LLM then human), how to tune thresholds for precision versus recall, and how to keep up with adversaries.

Personalizing an LLM is a retrieval and memory problem, not a per-user fine-tune. Learn how to assemble user context at request time, manage long-term memory without bloating the prompt, and respect privacy and the right to be forgotten.

A writing assistant has to feel instant while a cursor moves through a live document. Learn how to split fast deterministic checks from LLM rewrites, stream suggestions, and keep edits stable so the text does not flicker as the user types.

Auto-resolving support tickets means grounding answers in your knowledge base, safely calling real APIs (refunds, account changes), and knowing when to hand off to a human. Learn the agent loop, the guardrails, and how to measure resolution without breaking trust.

Long-running agents that loop, call tools, and spawn sub-tasks need durable execution, not a single request handler. Learn how to checkpoint state, recover from failures, bound runaway loops, and observe every step a non-deterministic agent takes.

Experimenting on LLM features is hard because the outputs are open-ended and quality is fuzzy. Learn how to assign traffic, pick metrics that are not just engagement, handle variance from non-determinism, and avoid the traps that make a winning variant lose in production.

Offering RAG as a product means handling messy ingestion, multi-tenant isolation, and per-customer index freshness, all behind a simple API. Learn the ingestion and query planes, how to keep tenants isolated, and how to give customers eval and observability.

Live speech-to-speech translation is a latency race across three models (recognize, translate, synthesize) while audio keeps streaming in. Learn the streaming pipeline, how to commit partial results without flip-flopping, and the tradeoff between latency and translation quality.

A celebrity posts and 50M followers need a notification. The hard parts are not sending one message, they are fan-out strategy, deduplication, channel routing, and not flooding users. Here is the architecture that handles both the long tail and the viral spike.

A rider taps request and within seconds a nearby driver is assigned. The interesting problems are geospatial indexing of moving drivers, the matching objective (nearest is not always best), and handling the race where two riders want the same car. Here is how to build it.

In payments, a duplicate charge or a lost credit is not a bug, it is a financial incident. The design hinges on idempotency keys, double-entry bookkeeping, and an append-only ledger that can be audited and reconciled. Here is how money systems stay correct.

A cache is easy until you scale it across many nodes. Then you face consistent hashing, eviction policy, the thundering herd on a cache miss, hot keys, and how stale you can tolerate. Here is the design that holds up under real traffic.

Metrics, traces, and IoT data are append-heavy, time-ordered, and rarely updated. A general database handles this badly. The wins come from columnar layout, compression tuned for timestamps, downsampling, and retention. Here is the time-series design interviewers want.

Raw clickstream lands in object storage and somehow becomes clean tables, dashboards, and ML features. The design questions are table format, the bronze-silver-gold layering, batch versus streaming, and how you handle schema drift and late data. Here is the lakehouse blueprint.

Crawling a few pages is trivial. Crawling the web means a URL frontier with priority, politeness per host, dedup at billions of URLs, and defenses against spider traps and infinite content. Here is the crawler architecture that scales without getting your IPs banned.

Showing the top 10 is easy. Showing a player their global rank among 50M others, updated live, is the real challenge. The answer hinges on the right data structure, sharding, and approximate ranks for the long tail. Here is how leaderboards scale.

Find the 20 nearest restaurants within 5km, fast, for millions of users. A naive distance scan over every place is hopeless. The design is about spatial indexing, geohash or quadtree partitioning, and handling dense cities versus empty regions. Here is how proximity search works.

Background jobs fail, retry, and get redelivered. If a job sends an email or charges a card, running it twice is a real problem. The design is at-least-once delivery plus idempotent handlers, visibility timeouts, and dead-letter queues. Here is how to make jobs safe.

When a page loads, an ad slot is auctioned to dozens of bidders and a winner is chosen before the page finishes rendering, all in under 100ms. The design covers the auction mechanism, the brutal latency budget, budget pacing, and fraud. Here is how an exchange runs millions of auctions per second.

Taking a cloud-API POC into a bank or defense network kills your model provider, your judge API, and your vendor dashboards all at once. Learn the signed-bundle supply chain, local evaluation loop, and zero-egress observability that make an air-gapped LLM stack actually operable.