TL;DR: Public benchmarks are a weak proxy: training-data contamination, narrowness, and saturation mean leaderboard scores overstate real-world ability. Build a task-specific eval set from your own data, measure what your product needs, and use LLM-as-judge for scale only after calibrating it against humans (it has position, verbosity, and self-preference biases). Combine automated evals, human review, and online metrics; treat evaluation as ongoing, not a one-time gate.
How to approach it. Make the case that evaluation, not modeling, is usually the bottleneck for LLM products. Then dismantle the two crutches people lean on (public benchmarks and LLM judges) honestly, and lay out what a trustworthy eval looks like. The signal is skepticism plus a constructive method.
A strong answer. Why public benchmarks mislead.
- Contamination. Models train on web-scale data that may include the benchmark's test set, so a high score can reflect memorization, not capability. It is pervasive and hard to rule out, so leaderboard numbers are an upper bound, not a guarantee.
- Narrowness and saturation. A benchmark measures one slice; topping MMLU says little about your customer-support or extraction task. Popular benchmarks also saturate (everyone scores high), losing discriminative power.
- Mismatch. The benchmark's distribution is not your traffic. What matters is performance on your task, your users, your failure modes.
Why LLM-as-judge is unreliable but necessary at scale. Using a strong model to grade outputs scales human-quality judgment, but judges carry systematic biases: position bias (favor the first or second answer shown), verbosity bias (prefer longer answers), self-preference (favor outputs from the same model family), and plain inconsistency. So judge scores are signal, not ground truth: calibrate against human labels on a sample, randomize answer order, use explicit rubrics, and audit periodically.
What a trustworthy eval looks like.
- A task-specific eval set built from real or realistic production queries with labeled expectations, covering common cases and known failure modes. Nothing replaces evaluating on your actual task.
- Right metrics for the task: exact-match/accuracy for structured tasks; faithfulness/groundedness for RAG; rubric-scored quality (calibrated judge plus human spot-check) for open-ended generation; safety metrics (toxicity, jailbreak, hallucination rate).
- Layered methods: cheap automated metrics for fast iteration, calibrated LLM-judge for scale, human review for the cases that matter, and online metrics (thumbs, task success, deflection) as the ultimate truth, since offline wins do not always hold.
- Ongoing, with regression tracking. Gate changes on the eval set, grow it from production failures, and watch for drift.
The defensible framing: distrust leaderboard scores, treat LLM-judge as calibrated-against-humans signal, and invest in a task-specific eval set plus online measurement, because evaluation quality caps how fast and safely you can improve.
Key takeaways.
- Leaderboard scores are an upper bound, not proof: contamination, narrowness, and saturation all inflate them.
- LLM-as-judge has position, verbosity, and self-preference biases; calibrate against human labels and randomize order before trusting it.
- The foundation is a task-specific eval set grown from production failures, not any public benchmark.
- Online A/B is the arbiter; offline wins do not always survive contact with real traffic.
What interviewers probe next.
- "How would you detect benchmark contamination?" Test on held-out, freshly-created, or private data the model could not have seen; large drops vs the public benchmark suggest contamination or overfitting.
- "How do you make LLM-as-judge trustworthy?" Calibrate against human labels, randomize order, use explicit rubrics and reference answers, prefer pairwise comparison over absolute scores, and audit a sample.
- "No labeled data to start?" Bootstrap with synthetic examples and a small curated set, then grow from production logs and human review.
- "Offline looks great, online does not, why?" Distribution shift, an eval set that misses real traffic, or metrics that do not capture user value; online A/B is the arbiter.
Common mistakes.
- Trusting public-benchmark scores as proof of real-world ability.
- Using LLM-as-judge as ground truth without calibrating against humans or controlling for its biases.
- No task-specific eval set, so you cannot tell whether a change actually helped your product.
- Treating eval as a one-time gate instead of an evolving suite fed by production failures.