TL;DR: Reproducibility means pinning everything that determines the result: data and feature version, code commit, hyperparameters, environment, and random seeds, logged per run by an experiment tracker (MLflow or Weights & Biases). A model registry then versions the trained artifacts and governs promotion through stages (staging to production) with lineage back to the exact run that produced each model. Without this you cannot rebuild a production model, debug a regression, or pass an audit.
How to approach it. Split the problem cleanly into two systems: experiment tracking (capturing each run's full provenance and metrics) and the model registry (versioning artifacts and governing promotion). Then enumerate what must be pinned for true reproducibility. "Log the metrics" is the shallow answer; the deep one is the whole provenance chain, because data is half of what determines a model.
A strong answer. Experiment tracking. Every training run logs:
| What to pin | Why it matters |
|---|---|
| Data / feature version (DVC hash, Delta version, feature-store version) | Data is half the model; an unversioned dataset makes a run irreproducible no matter how clean the code. |
| Code commit | The exact training logic that ran. |
| Hyperparameters and config | Every knob that moves the metric. |
| Environment (pinned deps or container image) | A library bump silently changes behavior otherwise. |
| Random seeds | Makes stochastic training repeatable. |
| Metrics and artifacts | Eval results, plots, and the model file, tied to the run. |
With all of that captured, any run re-executes to the same result, and you can diff two runs to see which change moved the metric. Tools: MLflow, Weights & Biases, or equivalents.
Model registry. Trained models are versioned artifacts with lineage back to the run, and therefore back to the data, code, and config that produced them. The registry governs promotion: a model moves from staging to production through a gate (beats the incumbent on eval, passes fairness and guardrail checks, then shadow and canary), and you can roll back to a prior version instantly. It is the single source of truth for "what is in production and where did it come from."
Why it matters concretely: debugging a production regression is diffing the current model's lineage against the last good one; an audit (EU AI Act, SOC 2) requires showing exactly what data and code produced a deployed model; and rerunning or onboarding work is trivial when provenance is captured rather than reconstructed from memory.
Key takeaways
- Reproducibility is the full provenance chain (data, code, config, environment, seeds), not just logged metrics.
- Data and feature version is the most-skipped pin and the one that most often breaks reproducibility.
- The tracker captures the process; the registry governs the products, with lineage and one-click rollback.
- Provenance is what an EU AI Act or SOC 2 audit actually demands, so build it in before it is urgent.
What interviewers probe next.
- "What's the most-forgotten thing to version?" Data. Code is usually in git; the dataset or feature version is what people skip, and it makes runs irreproducible.
- "Experiment tracker vs model registry?" The tracker captures the process (runs, metrics, provenance); the registry manages the products (versioned models, stages, promotion, rollback).
- "How do you reproduce a 6-month-old model?" Pull its registry entry, follow lineage to the exact data version, code commit, config, and environment, then re-run. If any of those was not pinned, you cannot.
- "How does this connect to CI/CD?" The pipeline logs runs to the tracker, gates on metrics, and registers and promotes through the registry, so promotion is governed and auditable.
Common mistakes.
- Logging metrics but not the data version, so runs cannot be reproduced.
- No environment pinning, so a dependency bump silently changes results.
- Promoting models by copying files around instead of a registry with lineage and rollback.
- Treating reproducibility as a nice-to-have until an audit or a production regression makes it urgent.