TL;DR: First, you should never have been at 100%: ship behind shadow, then canary, then a ramped A/B with automated rollback on the live metric. Given the drop, the prime suspect is training-serving skew (features computed differently online than offline). Work the funnel: confirm the regression is real and significant, compare offline-vs-online feature distributions, then check label/feedback delay and the candidate pool, rolling back to the known-good model while you investigate.
How to approach it. Split the answer explicitly into prevention and diagnosis; interviewers are scoring whether you treat deployment as a process, not an event. State the default hypothesis up front: the offline metric and the online metric are measuring different things, with training-serving skew as the most common concrete cause.
A strong answer. Safe rollout, before this ever happens: shadow the new model on live traffic and log its scores without serving them, so you can compare its score distribution against the incumbent. Then canary to roughly 1% behind a feature flag, watching p99 latency and CTR with a guardrail that auto-reverts on a significant drop. Then a proper A/B at increasing allocation with a pre-registered metric and a sample-size/power calculation, so noise and the peeking problem do not fool you. Keep the previous model one flag-flip away the entire time.
Diagnosis, given the 2% drop:
- Is it real? Check the confidence interval, not the point estimate. A 2% relative move can be noise at low volume; size the test before concluding anything.
- Training-serving skew (most likely). Compare the feature vectors the model receives online against the offline training data: schema mismatches, different default or imputation logic, a feature computed from a batch table offline but a stale cache online, unit or encoding drift. A feature that is point-in-time correct offline but reads a laggy source online is the classic culprit.
- Metric mismatch. Offline AUC/loss optimizes ranking quality; CTR depends on calibration, position bias, the candidate pool, and latency. A slower model can lower CTR purely through added latency even with strictly better ranking.
- Feedback loop and label delay. The offline labels came from the old model's exposures; the new model surfaces different items whose true CTR you have not observed yet, so early online numbers are biased against it.
Roll back to the known-good model first to protect the metric, then reproduce offline by scoring real logged online feature vectors and checking they match what training assumed.
Key takeaways
- Never reach 100% blind: shadow, canary with a guardrail, then a powered A/B, with rollback one flag-flip away.
- Training-serving skew is the default suspect; diff online feature vectors against what training assumed.
- A 2% move is not real until a power and significance check says so.
- Early online metrics are biased against the new model because its own exposures are still unlabeled.
What interviewers probe next.
- "How do you detect skew automatically?" Log serving features and run distribution checks (PSI/KS) against the training set; alert on per-feature drift.
- "Latency as a cause?" Yes. Measure end-to-end serving latency in the canary; even 100-200ms can move engagement.
- "How long do you run the A/B?" Until the pre-computed sample size for your minimum detectable effect, using sequential or Bayesian methods if you must monitor early.
Common mistakes.
- Assuming a code bug when the cause is data/skew or a metric mismatch.
- Having shipped to 100% with no canary or rollback path.
- Reading a 2% move as real without a significance and power check.
- Forgetting the new model's exposures are unlabeled, so naive early online metrics are biased against it.