TL;DR: Monitor four layers: operational (latency, error rate, throughput), input data (schema and feature-distribution drift via PSI or KS), predictions (score distribution and confidence), and outcomes (the live business metric once labels arrive). Alert on sustained shifts across multiple windows, not single batches, and wire drift detection to retraining triggers. At fleet scale the hard part is doing this uniformly as a platform and keeping alert noise low.
How to approach it. Name the failure mode that ordinary service monitoring misses: a model can be up, fast, and error-free while quietly making worse predictions because the world drifted underneath it. Lay out the layers, then stress label delay, because it dictates which signals you can act on and when.
A strong answer. Four layers of signal, ordered by how fast they fire.
- Operational and service: latency (p50/p99), error rate, throughput, resource use. Standard SRE telemetry. Necessary but not sufficient, because a healthy service can serve a degraded model.
- Input data quality and drift: schema and type checks and null rates (catch broken upstream pipelines) plus per-feature distribution drift against a training reference using PSI or KS tests. This is the earliest warning and needs no labels.
- Prediction monitoring: the model's output and score distribution and confidence. A sudden jump in the share of positive predictions often flags data drift or a broken feature before any outcome confirms it.
- Outcome and performance: the real metric (accuracy, AUC, business KPI) once ground truth arrives. This is the truth, but it lags. Labels for churn or fraud land days to weeks later, so you cannot lean on it for fast alerting.
The layers form a detection ladder: the cheaper, label-free signals fire first and the authoritative one fires last.
Architecture for the fleet. One standard logging schema every model emits (features, prediction, model version, timestamp) into a central store. A monitoring service computes drift and aggregates per model and per version. Dashboards and an alerting layer sit on top. Make it a platform: one instrumentation standard all 100-plus models plug into, not 100 bespoke setups, so coverage is uniform and a new model is monitored by default the day it ships.
Drift triggers feed retraining. A sustained drift or performance-decay alert opens an investigation and can kick off the retraining pipeline behind a promotion gate, closing the loop instead of paging a human to retrain by hand.
Alert hygiene. Alert on sustained, significant shifts across multiple windows, not single-batch noise. Set per-model baselines and thresholds, route by severity so on-call is not buried, and suppress known-benign drifts (a seasonal feature, a planned upstream change). Data drift is not always actionable, so pair every alert with the context needed to triage it in one glance.
Key takeaways
- Four layers: operational, input drift, prediction shift, outcome. The first three fire without labels; the last is truth but lags.
- One instrumentation standard as a platform, so a new model is monitored by default rather than retrofitted.
- Alert on sustained multi-window shifts with per-model baselines, not single batches, or the team learns to ignore pages.
- Wire drift and decay alerts to a gated retraining pipeline to close the loop.
What interviewers probe next.
- "Data drift vs concept drift?" Distribution tests catch data drift directly. Concept drift (the input-to-label relationship changed) only surfaces in the outcome layer, which is why you still need labels eventually.
- "Labels are delayed weeks. What do you watch meanwhile?" Input drift, prediction-distribution shift, and proxy or leading business indicators.
- "How do you avoid alert fatigue?" Sustained-shift thresholds, per-model baselines, severity routing, and suppressing known-benign drifts.
- "Multi-version or canary?" Track metrics per model version so a bad deploy is attributable and rollback is clean.
Common mistakes.
- Only service monitoring (latency and errors), missing silent quality decay.
- Waiting on outcome metrics that arrive too late to prevent harm.
- Bespoke monitoring per model, so coverage is patchy and new models ship unmonitored.
- Alerting on every single-batch fluctuation, training the team to ignore alerts.