LLM-as-a-Judge
When outputs are open-ended (summaries, chat answers, generated code), there is no exact match to score against, so you use a strong LLM to grade them against a rubric. It scales evaluation far beyond human review, but it is a fallible proxy with known biases (position, verbosity, self-preference), so you calibrate it against human labels and design carefully. Applied-AI interviews probe it because evaluating generative output is the hard part of shipping LLMs, and 'we eyeballed it' does not scale.
TL;DR: For open-ended outputs there is no gold answer to match, so you use a capable LLM to grade responses against a rubric (is it faithful? correct? helpful?). This scales evaluation past what humans can review and gives a repeatable metric. But the judge is a fallible proxy with biases, it can favor the first option, longer answers, or its own style, so you write clear rubrics, control for known biases, and calibrate the judge against human labels before trusting it. It is the same fallible-proxy pattern as a reward model.
Why you need a judge
Classification has exact metrics; generation does not. How "good" is a summary, a chat answer, or generated code? Exact-match and overlap metrics (BLEU/ROUGE) miss meaning: a paraphrase that nails the answer can score near zero on ROUGE, and a copied-but-wrong sentence can score high. Human review is the gold standard but does not scale to thousands of outputs per release at roughly 30-60 seconds per judgment. LLM-as-judge uses a strong model to score outputs against criteria, giving a scalable, repeatable evaluation you can run on every change for cents per item in seconds.
Common forms: scoring against a rubric (rate faithfulness/correctness 1-5), pairwise comparison (which of two is better, as in reward models), and reference-based grading (does the answer match the gold answer or the sources?). Prefer pairwise when you can: models are far more consistent at "A or B?" than at calibrated absolute scores, and a 1-5 scale tends to pile up at 4.
The biases to control
The judge is a model, so it inherits model failure modes:
- Position bias. In pairwise grading it may favor the first (or second) option regardless of quality. Run each pair in both orders and keep the verdict only if it agrees with itself; the disagreement rate is itself a useful health metric.
- Verbosity bias. It tends to prefer longer, more confident answers even when not better. Normalize for length or penalize padding in the rubric.
- Self-preference. It may favor outputs in its own style or from its own family, so do not judge GPT-4 output with a GPT-4 judge if you can avoid it.
- Rubric sensitivity. Vague criteria give noisy scores; specific, decomposed rubrics with few-shot examples of each grade are far more reliable (the G-Eval idea: score one dimension at a time with explicit definitions).
Worked example: calibrating before you trust
You want to judge 5,000 RAG answers for faithfulness. First, hand-label 200 of them. Run the judge on those 200 and build a confusion matrix against your labels. Say the judge agrees on 174 of 200, with Cohen's kappa of 0.68 (substantial, above the ~0.6 bar you would want). Now inspect the disagreements: if 18 of the 26 are the judge passing answers a human marked unfaithful, your prompt is too lenient, so tighten the rubric ("any claim not supported by the cited passage is a fail") and re-measure. Only once kappa clears your bar do you let the judge grade the remaining 4,800. Skipping this step is how teams ship a metric that quietly correlates with answer length instead of correctness.
| Symptom | Likely cause | Fix |
|---|---|---|
| Pairwise winner flips when you swap order | position bias | judge both orders, count only agreements |
| Longer answers keep winning | verbosity bias | length-normalize or penalize in rubric |
| Scores all cluster at 4/5 | scale compression | switch to pairwise or binary pass/fail |
| High accuracy, low kappa | class imbalance | report kappa/balanced accuracy, not raw agreement |
Why interviewers probe this
Evaluating generative output is the genuinely hard part of shipping LLMs, and "we eyeballed a few" does not scale. A strong answer explains that open-ended output needs a judge for scale, then immediately raises the biases (position, verbosity, self-preference) and the discipline of calibrating against human labels with an agreement metric like kappa. The follow-up they hold in reserve is "how do you know your judge is any good?", which is where the calibration loop and confusion matrix earn the senior signal.
Common misconceptions
- "The judge's scores are ground truth." It is a fallible proxy; calibrate it against human labels and report agreement.
- "Any prompt works as a judge." Vague rubrics give noisy scores; specific, decomposed criteria are far more reliable.
- "Judges are unbiased." Position, verbosity, and self-preference biases are well documented; control for them.
- "Raw agreement is a good calibration metric." On imbalanced data it is inflated; use Cohen's kappa or balanced accuracy.
- "BLEU/ROUGE are enough for generation." Overlap metrics miss meaning; judges (and task metrics) capture quality better.
Key takeaways
- Open-ended outputs lack a gold match, so a strong LLM grades them against a rubric, scaling evaluation past human review.
- Forms include rubric scoring, pairwise comparison, and reference-based grading; prefer pairwise for consistency.
- Judges have biases (position, verbosity, self-preference); use clear rubrics and control for them.
- Calibrate the judge against human labels with kappa, inspect disagreements, and treat it as a fallible proxy, not an oracle.
Check yourself before an interviewer does. Answer from memory first.
You need to trust an LLM judge on 5,000 RAG answers. What do you do first?