Offline vs Online Evaluation
Offline evaluation scores a model on held-out data; online evaluation measures its impact on real users (via an A/B test). They often disagree: an offline win frequently fails to move the online metric, because offline data is a static proxy and the real world has feedback loops, distribution shift, and second-order effects. The discipline is to gate with offline evals (fast, cheap) and confirm with online tests (the truth). Applied-AI interviews probe it because shipping on offline metrics alone is a classic, costly mistake.
TL;DR: Offline evaluation scores a model on a held-out dataset (accuracy, precision/recall, nDCG); online evaluation measures its real impact on users through an A/B test (conversion, engagement, revenue). They frequently disagree: a model that wins offline often fails to move the online metric, because offline data is a static proxy while production has feedback loops, distribution shift, and behavioral effects the dataset cannot capture. The discipline: use fast, cheap offline evals to gate changes, and online tests to confirm real impact, the online result is the truth.
Two different questions
- Offline: "Is the model more accurate on data we already have?" Fast, cheap, repeatable, and the basis of eval-driven development. But it scores against a static snapshot.
- Online: "Does deploying this model improve the outcome we care about for real users?" Measured by an A/B test on live traffic. Slower and costlier, but it measures actual impact.
Why they diverge
The offline-online gap is one of the most important facts in applied ML. Common reasons an offline win fails online:
- Proxy mismatch. The offline metric (e.g. click prediction accuracy) is not the business goal (e.g. long-term engagement); optimizing the proxy can even hurt the goal.
- Feedback loops. The model's own outputs change user behavior and future data (see recsys feedback loops); offline data, collected under the old model, does not reflect this.
- Distribution shift. Live traffic differs from the offline set (time, mix, new users), and a static eval cannot see it.
- Second-order / system effects. Latency, presentation, and interaction effects that offline scoring ignores.
This is why "offline good, online bad" (or even "offline neutral, online great") is routine.
Worked example: when offline lies
A recsys team swaps in a ranker that lifts offline nDCG@10 from 0.41 to 0.44, a clean 7% relative gain on a year of held-out click logs. They ship it to a 10% A/B holdout. Two weeks in, click-through is flat and 14-day retention is down 1.2% at p < 0.05. What happened? The new ranker learned to surface high-CTR clickbait, which scores beautifully on logged clicks (the proxy) but trains users to distrust the feed (the goal). The offline set, collected under the old ranker, never contained the clickbait-heavy slates the new model produces, so the offline metric literally could not see the failure mode. The team rolls back, adds a dwell-time and return-visit term to the offline objective, and the next candidate that passes offline holds up online. The lesson is not "offline is useless." It is that the offline metric was the wrong proxy, and only live traffic exposed it.
The discipline
Gate with offline evals because they are fast and cheap, iterate, catch regressions, and avoid shipping obviously-worse models. Then confirm online because the A/B test is the only measure of real impact. Never ship a model on offline metrics alone, and never run an expensive online test on every tiny change, the two-stage gate-then-confirm is the efficient, safe pipeline.
| Offline eval | Online A/B | |
|---|---|---|
| Question | more accurate on past data? | better outcome for real users? |
| Cost / speed | minutes, cheap | days to weeks, traffic at risk |
| Measures | a proxy on a static snapshot | actual impact with feedback loops |
| Role | gate (iterate, catch regressions) | confirm (the truth) |
Why interviewers probe this
Shipping on offline metrics alone is a classic, expensive mistake, so this distinction is a core maturity check. A strong answer explains that offline is a static proxy and online measures real impact, names the reasons they diverge (proxy mismatch, feedback loops, distribution shift), and prescribes the gate-then-confirm discipline. The reserved follow-up: "your offline metric and online metric disagree, which do you trust?" The online result, every time, and then you fix the offline proxy so it predicts online next time. Recognizing that an offline improvement is a hypothesis to be confirmed online is the experienced stance.
Common misconceptions
- "An offline improvement means ship it." Offline wins frequently fail online; confirm with an A/B test.
- "Offline and online measure the same thing." Offline is a static proxy; online measures real-user impact with feedback loops.
- "Online testing replaces offline evals." Offline gates cheaply and catches regressions; online confirms, you need both.
- "The offline metric is the goal." It is usually a proxy; optimizing it can diverge from the business outcome.
Key takeaways
- Offline evaluation scores held-out data (fast, cheap proxy); online evaluation measures real-user impact via A/B.
- They often disagree due to proxy mismatch, feedback loops, distribution shift, and system effects.
- Gate changes with offline evals, then confirm real impact online, the two-stage discipline.
- Treat an offline win as a hypothesis to confirm online, not a decision to ship.
Check yourself before an interviewer does. Answer from memory first.
Your offline metric says ship it, but the online A/B disagrees. Which do you trust?