Design a real-time fraud detection system where fraud is under 1% of transactions.

TL;DR: Two layers: fast deterministic rules for known patterns plus an ML model for the long tail, scored inline in tens of ms. Do not optimize accuracy or even ROC-AUC. Pick the operating threshold from the dollar cost of false positives (blocked good customers) against false negatives (fraud loss), report PR-AUC, and handle imbalance with class weights, not naive resampling. Design for delayed, biased labels and an adversary who moves.

How to approach it. Clarify three things up front: the latency budget (scoring is inline with the authorization, often under 50 ms), the cost asymmetry (a blocked legitimate customer can churn, a missed fraud is a chargeback), and how labels arrive (chargebacks land days to weeks later). Then lay out the architecture and spend most of the time on imbalance and thresholding, which is where this question is won or lost.

A strong answer. Architecture.

transaction --> feature lookup (online store: velocity, device, history)
            --> rules engine (hard blocks/allows for known patterns)   [~ms]
            --> ML risk model (gradient-boosted trees) -> score         [~ms]
            --> decision: allow / step-up auth / block, from threshold
            --> async: human review queue + label capture (chargebacks)

A rules layer catches known fraud cheaply and gives explainable hard blocks (a compliance and appeals requirement, not just speed). The ML model handles the long tail. Features come from a low-latency online store: velocity counts (transactions per card per hour), device and IP signals, account age, amount versus the user's own history. Score inline within the budget.

Imbalance, the crux. With under 1% positives, accuracy is meaningless: a model that predicts "never fraud" scores 99%. ROC-AUC flatters too, because the huge true-negative pool drowns out false positives. Report PR-AUC and precision/recall at the chosen threshold. Handle imbalance with class weights or cost-sensitive loss (preferred: no data thrown away) rather than SMOTE, which fabricates synthetic points in a space where fraud is non-stationary. If you resample, do it inside the cross-validation fold, never before the split.

The decision step is the real test. Do not block at 0.5. Derive the operating point from a cost matrix: expected loss of a missed fraud against expected cost (and downstream churn) of blocking a good customer. Usually you do not binary-block at all. Route medium-risk to a step-up challenge (3-D Secure, 2FA), reserve hard blocks for high confidence, and auto-approve the rest.

Feedback and adversary. Labels are delayed (chargebacks) and biased: you never learn whether a blocked transaction was truly fraud, so the training set is selection-skewed. Build a review queue for human labels and hold out a small unblocked control to estimate true rates honestly. Fraud is adversarial, so patterns shift weekly. Monitor score-distribution drift and retrain on a fast cadence with drift triggers.

Monitoring. Track precision and recall at threshold, dollar loss, false-positive (good-customer) rate, latency p99, and score-distribution drift.

Key takeaways

• Report PR-AUC and precision at the operating threshold; accuracy and ROC-AUC both lie under 1% positives.
• Set the threshold from a dollar cost matrix, not 0.5, and use a step-up tier for the gray zone.
• Prefer class weights or cost-sensitive loss over SMOTE; if you resample, keep it inside the CV fold.
• Labels are delayed and biased: keep an unblocked control and a review queue to measure truth.

What interviewers probe next.

• "Precision or recall more important?" Depends on costs, but you usually cannot block aggressively without harming good customers, so tune to a precision target and route the gray zone to step-up auth.
• "Why GBMs over deep nets here?" Tabular features, strong performance, fast inference, and interpretability for review and appeals. Deep nets earn their keep on sequence or graph signals.
• "Delayed labels: how do you evaluate now?" Proxy metrics (review-queue hit rate, score drift) plus a labeled holdout. Accept that ground truth lags.
• "Graph or ring fraud?" Add graph features (shared devices and cards across accounts) to catch coordinated rings that point rules miss.

Common mistakes.

• Reporting accuracy or ROC-AUC and ignoring precision at the operating threshold.
• Defaulting to a 0.5 cutoff instead of deriving the threshold from costs.
• SMOTE or oversampling applied before the train/test split, leaking and inflating offline metrics.
• Ignoring label delay and adversarial drift, so the model silently decays.