TL;DR: Shadow runs the new model on live traffic without serving its results, so you compare it to production at zero user risk (ideal for ML, since you validate predictions and latency before trusting them). Canary serves it to a small traffic slice with a guardrail that auto-reverts on regression. Blue-green keeps two full environments and switches all traffic at once (instant rollback by switching back). Use shadow then canary for models, and keep the previous version one flip away.
How to approach it. Define each strategy by what it validates and how much risk it puts on users, then make the point that shadow is the one ML especially needs, because offline metrics do not guarantee online behavior and shadow lets you watch the new model on real traffic safely. Tie everything back to fast rollback, which is the property interviewers actually care about.
A strong answer.
| Strategy | User risk | What it validates | Rollback |
|---|---|---|---|
| Shadow (dark launch) | None (output discarded) | Skew, latency, prediction-distribution on real inputs | N/A, never served |
| Canary | Small slice (1-5%) | Live KPI and guardrails on real users | Auto-revert flag flip |
| Blue-green | All-or-nothing at cutover | Full env in green before switch | Instant switch back to blue |
- Shadow (dark launch). Mirror live requests to the new model alongside production, but discard its output and serve only the current model. Log the shadow model's predictions and latency and compare against production on real traffic, catching training-serving skew, latency regressions, and prediction-distribution shifts at zero user risk. This is uniquely valuable for ML because the failure modes (skew, drift, calibration) only surface on production-shaped inputs that offline metrics never see. The cost is running inference twice.
- Canary. Route a small fraction (1-5%) of real traffic to the new model behind a feature flag, monitor the live metric and guardrails (latency, error rate, business KPI) with statistical care, and ramp up if healthy or auto-revert on a significant regression. It limits blast radius to the canary slice. Use it after shadow confirms basic sanity, because canary does expose some users.
- Blue-green. Maintain two complete environments (blue is current, green is new). Deploy and validate green, then switch all traffic at once; rollback is instant by switching back to blue. Simple, with clean instant rollback, but the cutover is all-or-nothing (no gradual exposure) and you pay for two full environments.
Typical ML flow: shadow to validate the new model against production safely, then canary or ramped A/B to measure real impact on a slice, then full rollout, with the previous model version always one flag-flip away (registry-backed) for instant rollback. Pair it with monitoring (drift, latency, KPI) so a regression triggers the revert automatically rather than waiting on a human.
Key takeaways
- Shadow is the ML-specific tool: it exposes skew, latency, and distribution issues on real inputs at zero user risk.
- Canary is a safety mechanism (limit blast radius); an A/B test is a measurement mechanism (is it actually better). Do not conflate them.
- Rollback must mean flipping to a still-running prior version, not rebuilding and redeploying.
- Monitor ML signals (drift, calibration, prediction distribution) during rollout, not just service health.
What interviewers probe next.
- "Why is shadow especially good for ML?" It surfaces skew, latency, and prediction-distribution issues on real inputs without risking users, and offline metrics cannot reveal those.
- "Canary vs A/B test?" Canary is primarily a safety mechanism (limit blast radius, watch guardrails); an A/B test is a measurement mechanism (is the new model better, with power and significance). They overlap but answer different questions.
- "How fast is rollback?" Blue-green and flag-based canary roll back near-instantly; the key is the previous version stays deployed and registered, not rebuilt on demand.
- "What do you monitor during rollout?" Live KPI, p99 latency, error rate, and prediction-distribution drift, with an automated guardrail that reverts.
Common mistakes.
- Straight cutover to 100% with no shadow or canary, trusting offline metrics.
- Confusing canary (safety) with an A/B test (measurement), so neither is done properly.
- Rollback that requires a rebuild or redeploy instead of flipping to a still-running previous version.
- Not monitoring ML-specific signals (drift, calibration) during the rollout, only service health.