Design an evaluation framework for an ads ranking system.

This is the question to design the scoreboard, not the player. A strong answer separates offline gates (AUC, calibration, NDCG) from the online verdict (A/B with revenue, user, and advertiser guardrails), and adds counterfactual replay so you can trust a model before it ever serves a real auction.

Updated Aug 2026 · Grounded in real Applied AI Engineer interview loops and written to a senior-engineer editorial bar.