MLOps & ML Engineering

The naive answer is 'retrain weekly.' The senior answer is a hybrid: drift-triggered retraining with a max-staleness fallback, plus a gate that a fresh model must beat the incumbent. Here is how to reason about it.

A feature store is easy to define and easy to get wrong. The signal is explaining the exact bug it prevents (training-serving skew) and the point-in-time correctness that stops label leakage. Here is the full picture.

The trap is jumping straight to 'add more GPUs.' The signal is profiling first, then applying the cheap, no-retrain levers in the right order. Here is the diagnosis-then-optimize playbook for p99.

The trap is describing plain software CI/CD. The signal is the three things ML adds: data validation, a model-quality gate against a baseline, and versioning data plus model plus code together. Here is what is genuinely different.

At thousand-GPU scale, hardware failure is the norm, not the exception, and a run that cannot resume wastes weeks. The signal is checkpointing strategy, deterministic resume, and minimizing lost work. Here is the fault-tolerant design.

Reproducibility is what separates an ML platform from a pile of notebooks. The signal is tracking the full provenance (data, code, config, metrics) and a registry that governs promotion. Here is what to track and why each piece matters.

Deploying a model is not flipping a switch. The signal is knowing what each rollout strategy validates, why shadow is uniquely useful for ML, and keeping rollback one step away. Here is the comparison and when to use each.

Grid search is the wrong answer at scale. The signal is knowing why random beats grid, how Bayesian optimization and early-stopping schemes (Hyperband/ASHA) spend compute smartly, and how to parallelize without stragglers.

GPU spend is usually the biggest line item in an AI org, and this question checks whether you think in utilization, not just capacity. The signal is squeezing per-unit cost (utilization, right-sizing, spot, quantization) before scaling out.

The most common ML production failure is not a code bug, it is a quiet upstream data change. The signal is validating at ingestion (schema plus distribution), data contracts with producers, and failing loud instead of training on garbage.

You cannot improve an LLM app you cannot see into, and LLM observability is not service monitoring. The signal is tracing multi-step chains, capturing inputs/outputs/tokens/cost, and online quality signals, not just latency and errors.

Caching is one of the biggest LLM cost levers, but 'cache the response' is naive for a non-deterministic system. The signal is distinguishing prompt (prefix) caching from semantic caching and knowing when each is safe.

LLMOps is not MLOps with bigger models. The signal is the genuinely new surfaces (prompts as deployable artifacts, eval-driven development, often no training step) and treating prompts with the same version discipline as code.

Routing each request to the right model is one of the biggest LLM cost/latency levers in production. The signal is matching query difficulty to model capability and knowing when the cascade pattern beats a classifier.

Models decay silently as the world shifts, and 'monitor for drift' is too vague. The signal is the actual statistical methods and distinguishing data drift you can detect without labels from concept drift you often cannot.

A high accuracy number hides slice failures, brittleness, and silent data bugs. The signal is naming the test layers that catch what the headline metric can't. Here is the framework that gets scored highest.

A trustworthy experiment platform is far more than a 50/50 split. The signal is consistent assignment, exposure logging, statistical rigor, and guardrails that survive peeking and sample-ratio mismatch. Here is the design.

When a model regresses or an auditor asks 'how was this built,' guesswork is a failing answer. The signal is tracking the whole chain so any model is reproducible and any past decision is explainable. Here is the chain.

Not every prediction needs a low-latency endpoint, and defaulting to one burns money. The signal is matching the serving pattern to the latency and freshness requirement. Here is the decision and when you'd reverse it.

Models are only as good as their labels, and noisy annotation silently caps performance. The signal interviewers want is a measured quality process, not a louder collection effort. Here is the answer.

A bad model in production needs a fast, safe rollback, but ML rollback is trickier than code: the model is data and the truth signal lags. The signal is pre-defined criteria plus a previous version kept warm. Here is the answer.

AI systems get expensive fast, and 'the GPU bill is huge' is not actionable without attribution. The signal is breaking spend down by model, feature, team, and request, then naming the levers. Here is the answer.

Shadow deployment validates a new model on real traffic without exposing users to it. The signal is knowing outputs are served in parallel then discarded, and how it complements canary and A/B. Here is the answer.

ML workflows are multi-step DAGs (ingest, feature, train, eval, deploy), and orchestrators run them reliably. The signal is the DAG/scheduling model plus what's actually ML-specific: data deps, versioned artifacts, drift triggers, eval gates.

An ML platform is the internal system that lets many teams build and ship models reliably. The signal is naming the components (data, features, training, registry, serving, monitoring) and arguing why standardizing beats per-team reinvention, plus when not to build.

Champion-challenger is how you improve a production model without betting the business on a hunch: the live model versus candidates competing on real metrics. The signal is the ongoing-competition framing and disciplined, pre-committed promotion.

Monitoring an ML system is more than CPU and latency; the model can silently rot while the dashboard stays green. The signal is the four-layer taxonomy (operational, data, prediction, outcome) and using inputs as leading indicators because labels lag.

Same model, two answers: clean offline, ugly online. The culprit is almost always a feature computed differently in the two paths. Here is the diff-based hunt that localizes it to a single column.

'Which model is in prod right now, and how did it get there?' should be a one-second lookup with an audit trail. If it isn't, you have a rollback and compliance problem already. Here is the promotion pipeline.

A fraud model fed a feature an hour behind is half-blind, but recomputing everything in real time burns money you don't need to spend. Freshness is a per-feature decision on a real cost curve. Here is how to manage it.

'Why was I denied?' is a question regulators and customers will ask, and 'we retrained since then' is not an acceptable answer. Exact reproduction means versioning everything. Here is what 'everything' actually means.

If you wait for labels to measure accuracy, you find out a model broke a month after it broke. Production monitoring has to work before the truth arrives. Here is what you watch instead.

Automating retraining sounds like a convenience until an auto-retrained model silently ships worse. The gates are the whole point. Here is how to automate retraining without automating a regression into production.

A data engineer renames a column and three production models silently start scoring on garbage. Features are a shared contract, and changing them needs the same care as a breaking API change. Here is the discipline.

An alert that fires constantly gets muted, and a model that fails with no alert is worse. Good ML alerting is a design problem with the same principles as SRE, plus ML-specific twists. Here is how to get it right.

'Works on my machine' is a dependency and environment problem, and for ML it includes CUDA versions and exact library pins that change numerical results. Here is how to make the production environment match training.

Editing a prompt is a code change with no compiler and no unit test by default, so quality regressions ship invisibly. Treating prompts and models as testable artifacts is what separates a toy from a product. Here is the harness.

Every team reinvents a spreadsheet of training runs and then drowns in it. The interviewer wants the platform that ingests runs, params, metrics, and artifacts at high write volume, links them by lineage, and makes thousands of experiments comparable, which is a different system from a model registry.

A green build still means nothing for model quality. The signal is an eval harness that runs deterministically in CI, compares against a frozen baseline, and blocks the merge on regressions. Here is how to make it fast, stable, and trusted.

Promotion is the easy part. Governance is who is allowed to ship what, on whose sign-off, with what documented, and how a model gets retired. Here is the control plane an auditor actually asks for.

'Just set a seed' is the wrong answer. Reproducibility means pinning the environment, the data, the code, and the hardware-level nondeterminism, all at once. Here is the full checklist and where it leaks.

You cannot put a 2 TB dataset in git, and copying it per experiment bankrupts you. The signal is knowing how content-addressed versioning works and when file-level (DVC) versus branch-level (lakeFS) fits. Here is the decision.

Anyone can wire tasks into a DAG. The signal is the patterns that keep it correct under retries and backfills: idempotency, data-aware triggering, and the asset model. Here is what separates a flaky pipeline from a trustworthy one.

Manual rollback means minutes of damage while someone wakes up. The signal is defining the trigger signals, thresholds, and guardrails that revert automatically, without flapping on noise. Here is how to make the loop safe.

Saying 'route 5% to the canary' is the easy part. The signal is how you split traffic deterministically, gather enough signal to decide, and ramp on evidence rather than vibes. Here is the mechanics layer.

A platform that no one uses is wasted headcount; a platform built too early is premature. The signal is justifying the investment by leverage and designing a paved path teams adopt willingly. Here is the framing.

Four drift tests, four different assumptions. The weak answer lists them; the strong one knows which handles high-dimensional embeddings, which needs binning, and which gives a calibrated p-value. Here is how to choose.

Three drifts get casually lumped together, but they have different detectability and different fixes. The interview signal is knowing which you can catch without labels and which one genuinely demands a new model.

Retrain too often and you burn money and risk regressions; too rarely and the model rots. The strong answer is a layered trigger policy with guardrails, not a single cron job. Here is how to set the cadence.

You cannot wait for ground truth to tell you the model broke. Prediction-side monitoring catches failures in minutes, not weeks, if you know which output signals actually move first.

Classic SRE SLOs assume a request is right or wrong. ML predictions are probabilistic and labels lag, so naive uptime SLOs miss the failures that matter. Here is how to set SLOs that actually cover model quality.

ML incidents are slipperier than service outages: nothing crashed, the model was just wrong. The strong answer covers detection, mitigation, and a blameless postmortem that fixes the system, not the person.

Eyeballing a few outputs does not scale, and a prompt tweak that fixes one case quietly breaks ten. A real LLM eval pipeline is a versioned dataset, layered scorers, and a CI gate. Here is the architecture.

Production usage can become your best source of training data, or a self-reinforcing trap. The signal is knowing how to capture clean feedback and how to break the loops that quietly corrupt the model.

A Deployment and a Service will serve a model, but GPUs break every Kubernetes default: scheduling, probes, autoscaling signals, and rollouts. Here is the setup that survives production, and when KServe earns its complexity.