TL;DR: A higher metric on one test set can be noise. Compare on the same held-out examples (paired, lower variance), put a confidence interval on the difference with a paired bootstrap, or use McNemar's test for paired classification errors. Keep the test set untouched by tuning, correct for multiple comparisons, weigh effect size, and confirm the offline win with an online A/B before believing it.
How to approach it. Reject "B scored higher, ship B." Push three ideas: compare on the same examples (paired, lower variance), quantify the difference with a confidence interval rather than a point estimate, and use a test built for the data type. Then layer the practical caveats: a clean holdout, multiple-comparison control, and online confirmation.
A strong answer. The trap. Model B at 0.84 vs A at 0.83 on one test set may be entirely within sampling noise; on a small or peculiar set, the "winner" flips on the next sample. The question is whether the gap exceeds the variation you would expect from the luck of which examples landed in the test set.
Methods, all paired.
| Method | Use when | Gives you |
|---|---|---|
| Paired bootstrap | Any metric (AUC, F1, BLEU); my default | CI on the B minus A difference, plus effect size |
| McNemar's test | Two classifiers, same examples | p-value on whether disagreements are symmetric |
| Paired t-test / Wilcoxon | Per-example scores, assumptions hold | p-value on mean per-example difference |
The paired bootstrap is the general workhorse: resample the test set with replacement many times (say 1,000), recompute the difference on each resample, and form a CI. If the 95% interval for (B minus A) excludes 0, B wins at that level, and you get an effect size for free. McNemar's test is purpose-built for paired binary classification: build the 2x2 table of where the models agree and disagree, then test whether the off-diagonal disagreement counts are symmetric, which is exactly the relevant signal. The common thread is pairing: scoring both models on identical examples differences out example difficulty and cuts variance, so you can detect smaller true gaps with the same data.
Practical caveats.
- Untouched test set. If either model saw the comparison set during tuning, you measure overfitting, not generalization.
- Multiple comparisons. Sweeping many models or metrics inflates false positives; correct (Bonferroni or FDR) or revalidate the winner on fresh data.
- Effect size, not just p. A significant 0.1% gain may not justify the cost or complexity; report the magnitude and decide on value.
- Online is the arbiter. Confirm the offline win with an A/B on real traffic, because latency, calibration, and distribution shift make offline and online diverge.
Key takeaways
- Pair the comparison: same examples for both models removes example-difficulty variance.
- A confidence interval on the difference beats two point estimates and gives you effect size.
- McNemar for paired classification, bootstrap for any metric, and never reuse a tuning set.
- Significance gates the decision; an online A/B confirms it. Both, in that order.
What interviewers probe next.
- "Why paired rather than separate test sets?" Evaluating both on the same examples removes example-difficulty variance, so you detect smaller true differences with the same amount of data.
- "When McNemar vs bootstrap?" McNemar is built for paired binary classification and focuses on disagreement counts; the bootstrap is general and yields a CI on any metric difference.
- "Significant but tiny gain, ship it?" Weigh effect size against cost and risk; significance is necessary, not sufficient.
- "How does this connect to A/B testing?" Same logic at the system level: offline significance gates, online A/B with power and peeking discipline confirms real-world impact.
Common mistakes.
- Declaring a winner from a single point-estimate metric with no uncertainty.
- Comparing on different (unpaired) test sets and inflating variance.
- Reusing a tuning set for the final comparison, so you measure overfitting.
- Ignoring multiple comparisons, or shipping a significant-but-trivial gain without an online check.