TL;DR: Precision is how many flagged positives are right; recall is how many real positives you caught; F1 balances them. Accuracy is misleading under class imbalance. AUC-ROC summarizes ranking across all thresholds but can look great while the model is useless at the operating point, especially on rare positives, where PR-AUC is more honest. Choose the metric from the cost of each error type.
How to approach it. Give crisp definitions, then immediately tie them to a decision: which error is more expensive here. The failure mode being tested is reciting formulas without knowing when each metric deceives, so lead with the misleading cases.
A strong answer. For a binary classifier with TP/FP/TN/FN:
- Precision = TP/(TP+FP): of everything you flagged positive, how much was actually positive. It matters when false positives are costly (flagging a legitimate transaction as fraud annoys customers).
- Recall = TP/(TP+FN): of all real positives, how many you caught. It matters when misses are costly (missing a cancer case, missing actual fraud).
- F1 = harmonic mean of precision and recall: one number when you need balance under imbalance; the harmonic mean punishes a model that sacrifices one for the other.
Where they mislead:
- Accuracy is the classic trap: with 99% negatives, a model that always predicts "negative" scores 99% accuracy and is worthless. Never report accuracy alone on imbalanced data.
- AUC-ROC measures how well the model ranks positives above negatives across all thresholds, independent of any single cutoff. It is great for comparing models, but on heavy imbalance ROC can read 0.9 while precision at the threshold you actually deploy is terrible, because the false-positive-rate denominator is dominated by the huge negative class. For rare-positive problems, PR-AUC (precision-recall) is the more honest summary because it focuses on the positive class.
- F1 hides the precision/recall split and weights them equally, which is wrong when one error is far costlier; report the underlying pair, or use F-beta to weight recall over precision (or vice versa).
| Metric | Misleads when | Reach for instead |
|---|---|---|
| Accuracy | Class imbalance | Precision, recall, PR-AUC |
| AUC-ROC | Rare positives, FP-sensitive deploy | PR-AUC, precision at threshold |
| F1 | FP and FN costs differ sharply | F-beta, the raw precision/recall pair |
The defensible move: state which error costs more for this problem, choose the metric and operating threshold accordingly, and add calibration if the scores need to mean probabilities.
Key takeaways
- Pick the metric from the cost asymmetry between false positives and false negatives, not from habit.
- Accuracy is a trap under imbalance; AUC-ROC is a trap when positives are rare and the deploy threshold is FP-sensitive.
- PR-AUC tracks the positive class, so it stays honest where ROC flatters.
- Ranking quality (AUC) and probability quality (calibration) are separate; a model can ace one and fail the other.
What interviewers probe next.
- "ROC-AUC vs PR-AUC, when?" PR-AUC when positives are rare and you care about the positive class; ROC-AUC for balanced comparison or when the true-negative rate matters.
- "How do you pick the threshold?" From the cost ratio of FP to FN, or to hit a precision/recall target the business requires, not the default 0.5.
- "What is calibration and why care?" A model can rank well (high AUC) yet output miscalibrated probabilities; if a downstream decision uses the probability (expected value), calibrate with Platt scaling or isotonic regression.
- "Multi-class metrics?" Macro vs micro averaging, and why macro surfaces poor performance on minority classes.
Common mistakes.
- Reporting accuracy on imbalanced data and declaring victory.
- Treating a high ROC-AUC as proof the deployed model is good, ignoring precision at the operating threshold.
- Optimizing F1 when the two error types have very different costs.
- Never stating which error matters more, so the metric choice is arbitrary.