TL;DR: Validate data where it enters training and serving: schema checks (types, columns, nullability), distribution checks (ranges, null rates, category sets, drift versus a reference), and freshness/volume checks. Then fail loudly (block the pipeline or alert) instead of training/serving on bad data. Back it with data contracts so upstream schema changes are agreed, not discovered in production. Most ML outages are silent data problems, not code.
How to approach it. Lead with the premise that resonates in the room: in ML, behavior depends on data, so a silent upstream change (a renamed column, units switched, a join that started returning nulls) degrades the model with no code change and no error. The defense is validation at the boundary plus contracts. Then enumerate the checks and the fail-loud principle.
A strong answer. Why this is the dominant failure mode. A model is only as good as its inputs. Upstream teams rename columns, swap units (dollars to cents), introduce nulls, or alter a join, and your pipeline keeps running and your model keeps scoring, just worse. No exception, no crash, so it stays invisible until a business metric drifts days later. This is the most common ML production incident, and it is a data problem, not a code one.
Validate at ingestion (training and serving).
- Schema validation. Expected columns present, correct types, nullability honored, enums within the allowed set. Catches a renamed/dropped column or a type change immediately.
- Distribution / statistical validation. Value ranges (no negative ages, amounts within sane bounds), null rates within tolerance, category cardinality, and drift versus a training reference (PSI or KS per feature). This catches a units change or a quietly shifted distribution that passes every schema check.
- Volume and freshness. Row counts in an expected band (catch a half-loaded table) and recency (catch a stalled upstream job).
Tools: a validation library (Great Expectations, TFX Data Validation, Deequ) running these as a gate in the pipeline.
Fail loud. On a violation, block the run (do not train/serve on bad data) or alert and quarantine, rather than silently proceeding. A failed pipeline you notice beats a degraded model you do not. For serving, validate incoming features and keep a safe fallback (default, last-good value, or reject) instead of feeding garbage to the model.
Data contracts. The systemic fix: an explicit contract with upstream producers covering schema, semantics, SLAs, and a rule that breaking changes require coordination and versioning. Enforced in the producer's CI, breaking changes cannot ship unannounced, so a schema change is negotiated and tested rather than discovered when your model regresses.
Key takeaways.
- The dominant ML outage is silent bad data, not a crashing code bug, so it needs its own instrumentation.
- Schema checks catch structural breaks; distribution and drift checks catch semantic ones that pass schema.
- Fail loud: block training, fall back safely on serving, and never log-and-proceed on a violation.
- Data contracts enforced in the producer's CI move the fix to the source instead of your dashboard.
What interviewers probe next.
- "Schema check vs distribution check, why both?" Schema catches structural changes (column/type); distribution catches semantic ones (units swapped, values shifted) that pass schema but ruin the model.
- "How does this tie to training-serving skew?" The same validation on serving features (against the training reference) catches skew before it silently hurts predictions.
- "What is a data contract concretely?" A versioned agreement on schema/semantics/SLAs between producer and consumer, enforced in the producer's CI so breaking changes cannot ship unannounced.
- "Fail the pipeline or alert?" For training, block (do not produce a worse model); for serving, fall back safely and alert; never silently proceed on violated data.
Common mistakes.
- Assuming production failures are code bugs and not instrumenting data validation.
- Schema checks only, missing semantic changes (units, distribution shifts) that pass schema.
- Logging a warning and proceeding instead of failing loud or quarantining.
- No data contracts, so upstream schema changes surface as a degraded model days later.