A/B Testing

TL;DR: An A/B test randomly assigns users to a control and a variant and compares a chosen metric, the randomization is what makes the difference causal. The tricky part is validity: stopping early when you see significance (peeking) inflates false positives, you must compute the sample size/power beforehand, a sample-ratio mismatch (observed split not matching intended) signals a bug that invalidates the test, and network effects and novelty break naive setups. For ML, the A/B test is how you confirm that an offline improvement actually moves the real metric online.

Why randomize

Comparing "before vs after" or self-selected groups confounds the result with everything else that changed. Random assignment balances confounders in expectation, so a difference in the metric can be attributed to the change, this is what makes A/B testing a tool for causal impact, not just correlation.

The validity traps

Running the test is easy; running a valid one is where it goes wrong:

• Peeking. Checking results repeatedly and stopping at the first significant moment dramatically inflates false positives. Fix: pre-compute the sample size and use a fixed horizon, or sequential/Bayesian methods built for continuous monitoring.
• Underpowering. Too few users cannot detect a real effect (false negatives). Do a power analysis for the minimum detectable effect up front.
• Sample-ratio mismatch (SRM). If a 50/50 split comes out 48/52, something is broken in assignment/logging, and the test is invalid. Monitor and halt on SRM.
• Interference / network effects. In social or marketplace products, treating one user affects controls, breaking independence. Use cluster/geo randomization.
• Novelty/primacy. Early behavior differs from steady state; run long enough.
• Multiple comparisons. Testing many metrics inflates false "wins"; pre-register the primary metric and correct.

A/B testing for ML

Offline metrics (accuracy, precision/recall, nDCG) gate a model change, but they often do not predict online impact (see offline vs online). The A/B test is the truth: route a slice of real traffic to the new model and measure the business metric, with guardrail metrics to catch regressions. This is also the basis of champion-challenger model promotion. When you want to optimize while learning rather than just measure, multi-armed bandits adapt allocation.

Why interviewers probe this

Shipping on offline metrics alone is a classic mistake, and A/B testing is how you avoid it, so it is constantly tested for both ML and product. A strong answer explains why randomization gives causal impact, then the validity traps (peeking, power, SRM, interference) that quietly produce wrong conclusions, and frames the A/B test as the confirmation that an offline win is a real online win. That rigor is exactly what the role needs.

Common misconceptions

• "Just check until it is significant." Peeking inflates false positives; fix the sample size or use sequential methods.
• "A 49/51 split is fine." A sample-ratio mismatch signals a bug that invalidates the test.
• "Offline improvement means ship it." Offline gains often do not hold online; confirm with an A/B test.
• "User-level randomization always works." Network effects require cluster/geo randomization.

Key takeaways

• Random assignment makes the measured difference causal, the core of A/B testing.
• Validity traps (peeking, underpowering, sample-ratio mismatch, interference, novelty) produce confident wrong conclusions.
• For ML, A/B testing confirms an offline improvement actually helps online; use guardrail metrics.
• Bandits adapt allocation when you want to optimize while learning rather than just measure.