Telling a real improvement from a lucky run
Most prompt iteration is superstition: change something, try it once, feel better, ship. This lesson replaces that with the smallest honest process, which takes about an hour to set up and is the highest-return hour in the course.
TL;DR: One run tells you nothing, because output varies. Build a fixed set of twenty to fifty examples with known-good answers, score every change against all of them, and change one thing at a time. That is not a heavyweight process, it takes an afternoon, and without it you are not iterating, you are redecorating.
Where you are. Last lesson of the module. You can write a prompt, enforce its output, choose examples, and recognise the standard failures. This lesson is how you tell whether anything you do to it actually helps. It is the bridge to module 6, and the habit it builds is the one that most separates people who ship from people who fiddle.
The trap, stated plainly
The natural loop is: try a prompt, look at the output, change a word, look again, decide it is better, move on.
Every step of that is unreliable, and it is worth seeing exactly why.
You looked at one output. The same prompt produces different text each time. You compared one draw from before against one draw from after, and there is no way to tell an improvement from the variation you would have seen anyway.
You looked at the case you had in mind. It is almost always a case that already worked. The inputs that fail are the ones you did not think of.
You changed more than one thing. Two edits and an improvement tells you nothing about which edit mattered, or whether one helped while the other hurt.
You were not neutral. You wrote the change and you want it to have worked. On a judgement call about text quality, that is not a small effect.
The result is a prompt that has been edited thirty times, each edit believed to be an improvement, with no evidence that the current version is better than the fifth.
The smallest honest process
Four steps. Genuinely an afternoon to set up.
1. Build a fixed set of examples. Twenty to fifty inputs with the output you would accept for each. Draw them from real data, not invented cases, and deliberately include the awkward ones: missing fields, ambiguous inputs, things that should be refused, edge formats. This set is the single most valuable artefact you will build in a project's first week.
2. Decide what "correct" means, precisely enough to check. Sometimes exact match on an extracted field. Sometimes a set of properties: valid shape, every claim traceable to the source, under the length limit. Sometimes a judgement, which is the hard case and is module 6's subject. Write it down, because a definition kept in your head will drift as you go.
3. Score every change over the whole set. Not one input. All of them, before and after. You are looking for the aggregate number, and equally for which specific items changed, because that is where the information is.
4. Change one thing at a time. One edit, one score. This is slower per step and much faster overall, because you learn something at each step instead of accumulating unattributed changes.
Reading the result honestly
Two things to watch for, and both are common.
A small change is probably noise. Going from 41 to 43 out of 50 is not obviously an improvement. If a change matters it usually moves the number more than that, and if you find yourself reasoning hard about two points, the honest answer is that you do not know yet.
Look at what moved, not just the total. A change that fixes four cases and breaks three shows as plus one and is not a small improvement, it is a trade. Sometimes a good one. You cannot see that from the total, and this is the most common way a team ships a regression they measured.
When the set is wrong
One caution, because it fails quietly. A fixed set stops being useful when it stops resembling reality. Real inputs shift, new document types appear, users ask things nobody anticipated.
The habit that fixes it: when a real failure reaches production, add it to the set before you fix it. The set grows with the problems you have actually had, which is exactly the set of problems worth defending against. This is also how a prompt suite becomes a regression suite over time.
Do this before moving on
Build the set. Twenty examples, an hour, for whatever you are working on. If you have nothing in progress, use any repetitive text task with data on hand.
Then score your current prompt against all twenty and write the number down. That number is the first honest thing you will know about your system, and everything you do afterwards has something to be measured against. Most people who take this course will skip this exercise, and it is the one with the highest return in the whole module.
Go deeper
- Evaluation-driven development is this process taken seriously as a way of working, and the direct prerequisite for module 6.
- Offline versus online evaluation covers what your fixed set can and cannot tell you, which matters before you trust it too far.
- LLM as a judge is step 2 when correctness is a judgement rather than a match. It has real pitfalls, and they are covered there.
- Practice question: How would you set up an evaluation framework from scratch? is this lesson as an interview question, and it is asked constantly in applied AI loops.
- Practice question: How do you evaluate generative output quality? is the hard version of step 2, and the one that separates strong answers.
Key takeaways
- One run cannot distinguish an improvement from ordinary variation. Score a fixed set, before and after.
- Twenty to fifty real examples, weighted toward awkward cases, is enough to start and takes an afternoon.
- Change one thing at a time, and read which items moved rather than only the total.
- When a failure reaches production, add it to the set before fixing it. That is how the set stays honest.
Check yourself before an interviewer does. Answer from memory first.
You reword a prompt, run it once, and the output looks better. What have you learned?