TL;DR: Pick one primary metric and a minimum detectable effect, then compute sample size and runtime from your baseline rate, variance, and target power (usually 80%) at your significance level (usually 0.05). Randomize at the right unit, run to the planned sample size, and do not stop early on a favorable peek (it inflates false positives). Use sequential or Bayesian methods if you must monitor continuously, and watch for novelty effects and guardrail metrics.
How to approach it. Treat it as designing a valid experiment, not "split traffic and compare." Lead with the pre-registration discipline (one primary metric, MDE, sample size) because the question is really testing statistical rigor and resistance to p-hacking.
A strong answer. Before launch:
- One primary metric + hypothesis. State the metric (e.g. conversion) and H0 (no difference) vs H1 (treatment improves it). Pre-register it so you are not metric-shopping afterward.
- Minimum detectable effect (MDE). The smallest lift worth shipping; a smaller MDE needs more samples.
- Power and significance. Set alpha (Type I / false-positive rate, commonly 0.05) and power 1-beta (probability of detecting a real effect, commonly 80%). From baseline rate, variance, MDE, alpha, and power, compute the required sample size and therefore runtime. Skipping this is how teams run underpowered tests and conclude "no effect" when they just lacked samples.
- Randomization unit. Randomize at the unit that avoids interference and matches the metric: user-level usually, cluster or geo when there are network effects, since user-level randomization leaks across a social product.
During and after:
- Do not peek and stop early. Repeatedly checking and stopping when p<0.05 inflates the false-positive rate far above 5% (the peeking problem). Either run to the planned sample size, or use sequential testing or Bayesian methods built for continuous monitoring.
- Interpret honestly. A p-value is the probability of data this extreme if H0 were true, not the probability the change works; pair it with the effect size and confidence interval. Confirm guardrail metrics (latency, revenue, complaints) did not regress while the primary moved.
- Watch confounders. Novelty and primacy effects (users react to the change itself), seasonality, and multiple-testing inflation if you check many metrics (correct with Bonferroni or FDR).
When a clean A/B is impossible (you cannot randomize), fall back to quasi-experimental methods: difference-in-differences, synthetic control, or geo holdouts.
Key takeaways
- Pre-registration is the whole game: lock the primary metric, MDE, alpha, and power before any traffic flows.
- Sample size is computed, not guessed; an underpowered null is "we lacked data," not "no effect."
- Peeking and stopping early inflates false positives; use sequential or Bayesian methods if you must watch live.
- A primary-metric win is not shippable until guardrail metrics confirm nothing else regressed.
What interviewers probe next.
- "Why is peeking a problem?" Each look is another chance to cross p<0.05 by noise; the true false-positive rate balloons. Sequential and Bayesian methods correct for it.
- "How long to run?" Until the precomputed sample size, and at least a full cycle (often a week) to cover day-of-week effects.
- "Many metrics show p<0.05, which are real?" Multiple-testing correction (FDR / Benjamini-Hochberg) and validation on a fresh window; do not cherry-pick.
- "Can't randomize users (network effects)?" Cluster or geo randomization, or diff-in-differences.
Common mistakes.
- No power or sample-size calculation, then misreading an underpowered null as "no effect."
- Stopping the moment it looks significant (peeking), inflating false positives.
- Claiming the p-value is the probability the change works (it is not).
- Ignoring guardrail metrics and shipping a primary-metric win that quietly hurt latency or revenue.