Your churn model's AUC jumps from 0.71 to 0.93 after adding a 7-day rolling feature. What now?

Question

Accepted Answer

TL;DR: A 22-point AUC jump from a rolling feature is leakage until proven otherwise. The usual culprit is temporal leakage: the rolling window includes data from at or after prediction time. Enforce an as-of cutoff so every feature uses only data strictly before the label time, switch to a time-based split, and confirm the lift survives. Real gains survive a chronological split; leakage does not. How to approach it. Do not celebrate. Name the suspicion (temporal leakage) and the mechanism (the rolling feature peeking past prediction time), then lay out the checks in order: feature construction, split strategy, and a clean holdout that simulates deployment. Frame the whole thing as one question: what would make this number honest. A strong answer. A jump that large almost never reflects a genuinely better feature; it usually means the feature encodes the label. With a 7-day rolling aggregate the classic bug is computing the window relative to "now" (training time) instead of each row's prediction time. For a user who churned on day 5, the window then quietly includes post-churn behavior, or includes no behavior at all, and that absence is itself the signal. The fix is an as-of join: every feature for a prediction at time t may use only events with timestamp strictly less than t. Then change evaluation. K-fold with random shuffling leaks future into past, which is exactly how a leaky feature looks great offline. Use a time-based split: train on [0, tcut), validate on [tcut, tcut + h), and keep a final holdout that mimics the real prediction cadence. Re-run the model with and without the rolling feature under that chronological split. If AUC collapses back toward 0.71, the lift was leakage. If a plausible gain survives (not 0.93), the feature is real and you keep it. Run these leakage checks alongside the split fix: inspect feature importance (one feature that dominates everything is a tell), compare the feature's distribution for churned vs retained users near the label boundary, and confirm the feature is even computable at serving time from past data only. A 0.93 AUC on churn is suspicious on its face; honest churn models usually land far lower. Key takeaways A 20+ point jump from a rolling feature is a leakage hypothesis, not a win, until a chronological split confirms it. The root cause is windows anchored to training time instead of each row's prediction time; an as-of join with ts < t fixes it. Random k-fold is the wrong evaluator for temporal data; it hides exactly the leakage you are hunting. Sanity-check the absolute number: 0.93 churn AUC is implausibly high and should trigger scrutiny by itself. What interviewers probe next. "Other leakage sources besides time?" Target leakage (a feature derived from the label), train/test contamination (the same user in both splits), and preprocessing fit on the full dataset (a scaler or target encoder fit before the split). "How do you prevent this systematically?" A point-in-time-correct feature store with as-of joins, and time-aware CV as the default for any temporal problem. "AUC limitations here?" Under heavy imbalance, AUC-ROC can look strong while the model is useless at the operating threshold; report PR-AUC and calibration too. Common mistakes**. Treating the jump as success and shipping it. Defending it with random k-fold numbers, the very split that hides temporal leakage. Fixing the split but leaving the feature computed relative to training time rather than each row's prediction time. Ignoring that a 0.93 churn AUC is implausible on its face.