CI/CD for Models

TL;DR: Software CI/CD tests code; ML CI/CD must also test data and the model, because behavior depends on data, not just code. The pipeline runs data tests (schema, distributions, no leakage), model tests (meets a metric threshold, beats the current production model, and per-slice performance), and behavioral tests (the model behaves correctly on targeted cases), then gates deployment on all of them, with canary/shadow rollout and rollback. "We tested the code" is not enough; the data and model gates catch the failures that would otherwise reach users.

Why ML CI/CD is different

In software, the same code gives the same behavior, so testing the code suffices. In ML, the data is half the system: a change in an upstream feature, a retrain on shifted data, or a leak can change behavior with the code untouched. The classic incident is a unit-test-green deploy that tanks precision because an upstream team started sending an amount field in cents instead of dollars. No code changed, every assertion passed, and the model silently learned the wrong scale. So the pipeline must validate data, model, and code, and gate on a quality bar, not just "tests pass."

The three layers of tests

• Data tests. Validate inputs: schema/types, null rates, value ranges, freshness, and no leakage. A silent upstream data change is a top failure source; catch it before it reaches the model. Great Expectations or a few pandera schemas turn "the cents bug" into a hard CI failure (see feature store for training-serving consistency).
• Model tests. The model must meet a minimum metric and beat the current production model on a held-out, time-correct set, not just "improved over nothing." Check per-slice performance so a subgroup is not failing under a good aggregate (see eval-driven development).
• Behavioral tests. CheckList-style: invariance (a paraphrase or an irrelevant name swap should not flip the prediction), directional (a known change moves the score the expected way), and minimum-functionality (must-pass simple cases). These catch brittleness an aggregate metric hides.

Worked example of the gate logic, expressed as the assertions CI runs before any deploy:

def gate(cand, prod, slices):
    assert cand.auc >= 0.90, "below absolute floor"
    assert cand.auc >= prod.auc - 0.002, "regresses vs production"
    for s in slices:                       # e.g. region, new-vs-returning user
        assert cand.auc_by[s] >= 0.85, f"slice {s} below floor"
    assert cand.invariance_pass and cand.mft_pass, "behavioral failure"
    return True   # all three layers green -> eligible to promote

The small tolerance (prod.auc - 0.002) matters: held-out metrics have noise, and a strict "must strictly exceed" gate blocks neutral-but-safe refactors forever.

Gating, rollout, and rollback

All three gates must pass before deploy; the model is promoted through the registry. Then use canary/shadow rollout (the new model on a small slice, say 5 percent of traffic, or in parallel with no user impact) and confirm with guardrail metrics before full traffic, with instant rollback if it regresses. Offline gates are fast and cheap; the online A/B test is the only thing that confirms real impact, since offline AUC and downstream revenue do not always move together.

Why interviewers probe this

Shipping ML safely is a maturity signal, and "how do you deploy a model change without breaking things?" expects more than software CI/CD. A strong answer adds data tests (the ML-specific failure source), model gates (threshold + beats-baseline + per-slice), and behavioral tests, then gated promotion with canary/shadow and rollback. Recognizing that data, not just code, must be tested is the core insight; naming the noise tolerance on the beats-baseline gate is the senior tell.

Common misconceptions

• "Testing the code is enough." ML behavior depends on data; you must test data and the model too.
• "A model that improved should ship." It must beat the current production model (within noise) and pass per-slice checks, not just improve over nothing.
• "Accuracy is the only model test." Per-slice and behavioral (invariance/directional) tests catch failures aggregate metrics hide.
• "Deploy straight to all traffic." Use canary/shadow and confirm with guardrails before full rollout; keep rollback ready.

Key takeaways

• ML CI/CD tests data, model, and code, because behavior depends on data, not just code.
• Data tests (schema/distribution/leakage), model tests (threshold + beats baseline + per-slice), and behavioral tests gate deployment.
• Promote via the registry with canary/shadow rollout and instant rollback.
• "We tested the code" is insufficient; the data and model gates catch user-facing failures.