Eval-Driven Development and Golden Datasets
You cannot improve an LLM system you cannot measure, so the first thing to build is an evaluation: a golden dataset of representative inputs with expected behavior, plus metrics, that you run on every change. This turns 'it feels better' into a number, catches regressions before users do, and lets you iterate quickly. Applied-AI interviews probe it because teams that ship reliable LLM features evaluate continuously, and 'we tried some prompts and it looked good' is the anti-pattern.
TL;DR: Build the evaluation before you optimize. A golden dataset of representative inputs with expected outputs or behaviors, paired with metrics (exact-match, task metrics, or LLM-as-judge for open-ended outputs), lets you turn "it feels better" into a number and run it on every change. This catches regressions before users do, makes iteration fast and objective, and is what separates teams that ship reliable LLM features from those that tweak prompts and hope. Treat evals like a test suite for your AI system.
You cannot improve what you cannot measure
LLM systems are stochastic and open-ended, so "we tried some prompts and it looked good" gives you no way to know whether a change helped, hurt, or just moved things around. The same prompt at temperature 0.7 can pass on Monday and fail on Tuesday. Eval-driven development flips this: define how you will measure quality first, then optimize against it. The eval becomes the source of truth for every prompt change, model swap, retrieval tweak, or fine-tune. The discipline is identical to writing the test before the feature.
The golden dataset
A golden dataset is a curated set of representative inputs with expected outputs or behaviors:
- Representative. Cover the real distribution, including hard cases, edge cases, and known failure modes (not just easy happy-path examples). A common mistake is a set that is 90% trivial queries, where a regression on the 10% that matters goes unseen.
- Expected behavior. Exact answers where possible; for open-ended outputs, reference answers or rubric criteria for an LLM judge.
- Living. Grow it over time, every production failure becomes a new test case, so the eval gets stronger and prevents regressions of past bugs.
Pair it with the right metrics: exact-match/F1 for structured tasks, retrieval recall and faithfulness for RAG, and judge scores for generation quality.
Worked example: sizing the set and reading the result
Suppose you run a support-bot eval of 200 cases. The baseline prompt passes 170 (85%); a new prompt passes 178 (89%). Is +4 points real or noise? With n=200 and p≈0.87, the standard error on a proportion is about sqrt(0.87 * 0.13 / 200) ≈ 0.024, so a 95% band is roughly ±4.7 points. The +4 is inside the noise, so do not ship on it. To resolve a true 4-point difference you need closer to 600-800 cases, or a paired comparison that scores both prompts on the same items and counts only the cases that flipped (McNemar's test), which has far more power. This is why "it passed more on my 12 hand-picked examples" tells you nothing.
| Set size | 95% margin on ~85% pass rate | Smallest real difference you can detect |
|---|---|---|
| 30 | ±13 pts | only large swings |
| 200 | ±5 pts | ~5+ pts, paired better |
| 800 | ±2.5 pts | ~2-3 pts |
Tooling to name: promptfoo or OpenAI Evals for the harness, run as a CI job (GitHub Actions) on every PR; LangSmith or Braintrust for tracing and dataset versioning; store the dataset in git so a diff to the eval set is reviewed like code.
What it buys you
- Objective iteration. Every change gets a number, so you keep what helps and revert what hurts.
- Regression protection. Run the eval as a gate in CI so a "small tweak" cannot silently break something that worked. Block merge if the pass rate drops below baseline minus the noise band.
- Speed. Fast offline iteration (minutes, cents) before the slower, costlier online A/B test confirms real impact.
Offline evals gate; they do not replace the online A/B test that confirms real-world impact.
Why interviewers probe this
Continuous evaluation is the practice that most distinguishes teams who ship reliable LLM features, and its absence ("we eyeballed it") is the clearest red flag. A strong answer says you build the eval first, describe a representative, living golden dataset with appropriate metrics (including LLM-as-judge for open-ended output), and run it as a regression gate on every change, then confirm online with an A/B test. The follow-up they hold in reserve is "how do you know a 3-point gain is real?", which is where sample size and paired tests separate the senior answer from the junior one.
Common misconceptions
- "Eyeballing a few outputs is enough." It does not scale or catch regressions, and a dozen examples cannot distinguish a real gain from noise.
- "Build the eval after the model works." Build it first, you cannot improve what you cannot measure.
- "A static eval set is fine forever." Grow it, turn each production failure into a new test case.
- "Any bump in pass rate means ship." Check it against the sample-size noise band first; small gains on small sets are usually noise.
- "Offline evals replace A/B tests." They gate quickly; the A/B test confirms real online impact.
Key takeaways
- Build the evaluation before optimizing: a golden dataset of representative inputs plus the right metrics.
- It turns "feels better" into a number, catches regressions, and speeds iteration.
- Size the set to the effect you care about; a few hand-picked examples cannot tell signal from noise.
- Make it living: every production failure becomes a new test case.
- Offline evals gate changes; an online A/B test confirms real impact.
Check yourself before an interviewer does. Answer from memory first.
A new prompt passes 178 of 200 eval cases versus the baseline's 170. Do you ship it?