TL;DR: Predict P(click | user, ad, context) with a feature-rich model (gradient-boosted trees or a deep model like DLRM or DCN), serve it in real time within a tight latency budget, and keep the probability calibrated, because ad ranking multiplies pCTR by bid (expected value), so a miscalibrated score directly mis-prices the auction. Handle massive sparse categorical features, delayed and biased click labels, the feedback loop, and freshness. Evaluate with AUC and log-loss offline, revenue and CTR online.
How to approach it
Stress what makes CTR different from a generic classifier: the output feeds an auction (expected value = pCTR x bid), so calibration matters as much as ranking. The scale (billions of events, high-cardinality sparse features) and latency (score many candidate ads per request in milliseconds) shape the design. Then lay out features, model, serving, and the data-bias issues. Lead with calibration; that is the line on the rubric most candidates miss.
A strong answer
Problem framing. For each ad request, score candidate ads by predicted click probability, then rank by expected value (pCTR x bid, plus quality and relevance terms) for the auction. Because both ranking and pricing use the actual probability, the model must be well-calibrated: a predicted 2% must really mean about 2%, or you over- or under-price and lose money. That is why CTR is a calibrated probability problem, not just a ranking one.
| Concern | Why it bites | Lever |
|---|---|---|
| Calibration | pCTR x bid prices the auction | log-loss training, isotonic/Platt, monitor in prod |
| Sparse features | millions of ids | embeddings, learned crosses (DCN/DeepFM) |
| Delayed labels | clicks/conversions arrive late | delayed-feedback model, attribution window |
| Feedback loop | train only on shown ads | exploration, propensity weighting |
| Latency | hundreds of ads in a few ms | online feature store, precomputed embeddings |
Features. Massive and largely sparse categorical: user (history, demographics, embeddings), ad (advertiser, creative, category), context (placement, device, time), and cross features (user x ad-category interactions, which carry much of the signal). High cardinality (millions of ids) means embeddings, and feature crosses are central, which is why DCN and DeepFM exist to model them automatically.
Model. Gradient-boosted trees are a strong, calibratable baseline for tabular features. At the largest scale, deep models (Wide and Deep, DLRM, DCN, DeepFM) handle huge sparse embeddings and learn feature interactions. Whatever the model, add a calibration step (Platt or isotonic, or train with log-loss which is a proper scoring rule) and monitor calibration in production.
Serving. Candidate ads come from earlier retrieval and targeting; the CTR model scores hundreds per request within a few ms, so use an online feature store for low-latency features, precompute embeddings, and batch the scoring. The pipeline is retrieval, then CTR ranking, then auction.
Data realities (where it gets hard):
- Delayed and biased labels. A click may arrive later (a conversion much later), and you only observe outcomes for ads you showed, so the data is biased by the old model's choices, position bias included. Correct with position-aware modeling and exploration or propensity weighting.
- Feedback loop. Training on shown-ad clicks reinforces what you already serve, so add exploration to give new ads impressions.
- Freshness. Ad performance and trends shift fast, so retrain frequently (often near-real-time or online updates) and monitor drift.
Evaluation. Offline: AUC for ranking and log-loss plus calibration for probability quality. Online: A/B on revenue, CTR, and advertiser and user outcomes, since offline rarely captures auction dynamics.
Key takeaways
- Calibration, not just AUC, decides revenue: a miscalibrated pCTR mis-prices every auction even when ranking looks fine.
- High-cardinality sparse features drive the architecture choice toward embeddings and learned crosses (DCN, DeepFM, DLRM).
- The training data is biased by what you showed; counter with position-aware modeling, propensity weighting, and explicit exploration.
- Score hundreds of candidates in a few ms via an online feature store and precomputed embeddings, then rank by expected value.
What interviewers probe next
- "Why does calibration matter more here than in many classifiers?" The probability is multiplied by bid to price the auction; miscalibration directly mis-ranks and mis-prices and costs revenue, even if ranking AUC looks fine.
- "Position bias?" Higher slots get more clicks regardless of relevance, so model position explicitly (or use it only at train time, or inverse-propensity weight) so you do not learn that slot 1 means good.
- "Why GBM vs deep?" GBMs are strong, fast, and calibratable on tabular features; deep models win at the largest scale with huge sparse embeddings and learned crosses.
- "Delayed conversions?" Use delayed-feedback modeling or wait-and-attribute windows; do not treat a not-yet-converted impression as a hard negative.
Common mistakes
- Treating CTR as pure ranking and ignoring calibration, then mis-pricing the auction.
- Ignoring position bias and the show-only feedback loop, baking the old model's choices into the new one.
- No exploration, so new ads never accrue data.
- Evaluating with AUC alone and missing calibration, log-loss, and online revenue effects.