TL;DR: ML CI/CD keeps everything standard CI/CD does (lint, tests, build, deploy) and adds three things, because the model is a function of data, not just code: validate incoming data (schema, distribution), gate on model quality (a retrained model must beat the incumbent on a held-out, time-correct eval, not merely pass unit tests), and version data + features + model + code together so any run is reproducible. Deployment uses shadow then canary because offline metrics never guarantee online behavior.
How to approach it. Acknowledge the shared base in one sentence, then spend your air on what is ML-specific. The framing that scores: in software, behavior is determined by code; in ML it is determined by code and data, so the pipeline has to test, gate, and version the data path too. Everything else follows from that.
A strong answer. Standard CI/CD still applies in full: lint, unit tests on the code, build artifacts, deploy. ML bolts three things on top, plus a staged deployment discipline.
- Data validation as a gate. Before training or serving, check schema, types, null rates, and feature-distribution sanity against expectations. A silent upstream data change (a column that goes all-null, a unit that flips from cents to dollars) is the single most common ML outage, and ordinary tests never see it because the code is untouched. Run the checks in CI on a sample.
- Model evaluation as a release gate. This is the decisive difference. A build that compiles and passes every unit test can still be a worse model. So the pipeline retrains (or loads the candidate), evaluates on a held-out, time-correct eval set, and compares to the current production model. Promote only if it meets or beats the incumbent on the chosen metrics plus fairness and guardrail checks. There is no analog in software CI: passing tests proves correctness, not quality.
- Versioning data + features + model + code together. Reproducibility means pinning the dataset and feature version and the model artifact alongside the code commit (DVC, MLflow, or a Delta version plus a model registry), so any production model can be rebuilt, audited, and a regression traced to exactly what changed.
- Deployment with shadow/canary and rollback. Offline metrics do not predict online behavior (training-serving skew, latency, distribution shift), so promotion runs through shadow then canary with a guardrail that auto-reverts, never a straight cutover.
The mental model: software CI/CD is a subset. ML CI/CD adds a parallel data and model track that runs alongside the code track and joins it at the release gate.
Key takeaways
- The eval-vs-baseline gate is the part with no software equivalent: green tests do not mean a better model.
- A silent data change, not a code bug, is the most common ML outage, so data validation is a first-class CI step.
- Reproducibility requires versioning data and features alongside code and the artifact, not code alone.
- Promotion is staged (shadow then canary) with an automatic guardrail, because offline numbers do not bind online behavior.
What interviewers probe next.
- "Why isn't passing unit tests enough?" Unit tests check code correctness; they cannot tell you the model got worse. You need an eval-vs-baseline gate.
- "What do you version and why?" Data and feature version, model artifact, and code commit, so a result is reproducible and a bad model is attributable to a specific change.
- "How do you test a non-deterministic model in CI?" Assert metrics within tolerance on a fixed eval set rather than exact outputs, and pin seeds where you can.
- "Continuous training?" Automated retraining triggered by schedule or drift, feeding the same eval gate and staged rollout, so the loop is hands-off but still safe.
Common mistakes.
- Describing plain software CI/CD and missing data validation and the model-quality gate.
- Promoting a model because the pipeline "passed" without comparing it to the incumbent.
- Versioning only code, so a production model cannot be reproduced or audited.
- Straight-to-prod deploys with no canary, trusting offline metrics.