ab testing

Offline metrics gate releases on a fixed dataset, online metrics show what users actually did, and offline wins routinely fail to hold. Why you need both.

Shipping a model is an experiment, and this question separates people who run A/B tests from people who p-hack them. The signal is pre-registering the metric, sizing the test, and resisting the urge to peek.

Hypothesis testing underlies every experiment, and the signal is interpreting the p-value correctly (it trips most people) and picking the right test. Here is the rigorous, plain-language answer.

Bandits optimize while they learn, shifting traffic toward winners mid-experiment instead of fixing it like an A/B test. The signal is the explore-exploit tradeoff, the three core algorithms, and knowing exactly when a bandit beats a clean A/B test.

Running an A/B test is easy; running a valid one is hard. The signal is naming the traps that produce confidently wrong conclusions and the fix for each. Here is the checklist a senior experimenter carries.

A pre-existing gap between your groups means randomization or instrumentation is broken, and the whole experiment is suspect. Strong candidates name sample-ratio mismatch on sight. Here is the full diagnosis and the disciplined response.

For search and recommendation, A/B tests can be slow and noisy because they compare different users. Interleaving compares two rankers within the same user's results and detects winners with far less traffic. Here is how.

Separate offline gates (AUC, calibration, NDCG) from the online verdict (A/B with revenue and advertiser guardrails), plus counterfactual replay between.

Experimenting on LLM features is hard because the outputs are open-ended and quality is fuzzy. Learn how to assign traffic, pick metrics that are not just engagement, handle variance from non-determinism, and avoid the traps that make a winning variant lose in production.

A trustworthy experiment platform is far more than a 50/50 split. The signal is consistent assignment, exposure logging, statistical rigor, and guardrails that survive peeking and sample-ratio mismatch. Here is the design.

Shadow deployment validates a new model on real traffic without exposing users to it. The signal is knowing outputs are served in parallel then discarded, and how it complements canary and A/B. Here is the answer.

Champion-challenger is how you improve a production model without betting the business on a hunch: the live model versus candidates competing on real metrics. The signal is the ongoing-competition framing and disciplined, pre-committed promotion.