Examples and reasoning steps, and when each stops paying
Two techniques get reached for reflexively and both have a point where they stop helping and start costing. This lesson covers what each actually does, how to choose examples that carry information, and the reasoning trap that costs teams real money.
TL;DR: Examples teach shape and edge cases, and three well-chosen ones beat ten similar ones. Asking a model to reason step by step helps on genuinely multi-step problems and does nothing on lookup or classification, where it just costs tokens and latency. Both are tools with a range, and using them outside it is the most common way a prompt gets slower and worse at the same time.
Where you are. You can assemble a prompt and get output your code can trust. Two levers get pulled next more than any others. Both work. Both are over-applied, and the over-application is expensive in ways that do not announce themselves.
What an example actually does
Putting worked cases in the prompt is the highest-value-per-token move available, and it is worth being precise about why, because the reason tells you which examples to pick.
An example communicates three things at once, and two of them are hard to write as instructions:
- Shape. The exact form of a good answer, more precisely than a description.
- Edge handling. What to do in the awkward case, shown rather than explained.
- Register. How terse, how formal, how hedged.
The second is where most of the value hides. Describing what to do when a document is ambiguous takes a paragraph and is still vague. Showing one ambiguous document and the correct output is unambiguous.
Choosing examples that carry information
The instinct is to add examples that look like the typical case. That is close to useless, because the typical case is what the model already gets right.
Pick examples at the boundaries. The document with a missing field. The ticket that is two complaints in one. The input in another language. The case where the right answer is "not applicable". Each one buys you a behaviour you would otherwise have to describe and hope.
Three is usually enough, and ten is usually worse. Diminishing returns arrive fast, and every example is tokens on every request, forever. Ten near-identical examples cost ten times as much as one and teach little more.
Make the examples correct. Obvious, and violated constantly. An example with a subtly wrong output teaches the model to be subtly wrong, and it will be reproduced faithfully. Examples are training signal for the request; treat them like code, not like documentation.
Watch for order effects. If all your examples of one category come last, that category can become the default. Mix them.
When step-by-step reasoning helps
Asking a model to work through a problem before answering measurably improves accuracy on some tasks. The word "some" is doing heavy lifting.
The test is simple: would a careful person need to work through intermediate steps to get this right? If yes, giving the model room to do the same helps. If the answer is a lookup or a judgement made at a glance, it does not.
Sentiment classification does not become more accurate because the model wrote three sentences first. It becomes slower, more expensive, and more likely to talk itself out of a correct first instinct.
The trap worth naming
There is a specific failure that costs teams real money, and it is worth internalising before you meet it.
A model's stated reasoning is not a record of how it reached the answer. It is more text, generated the same way as the answer. It usually correlates with the actual process, and it can be a fluent, convincing justification for a conclusion arrived at otherwise.
Two consequences, both practical:
- Do not treat the reasoning as an explanation you can show a user as justification. It reads like one. It is not verified to be one.
- Do not debug by reading it. If a classifier is wrong, the reasoning will explain confidently why the wrong answer was right. Debug against inputs and outputs.
Newer models that do extended internal reasoning shift where this happens but not whether it happens. The output is still generated text.
The decision, compressed
- Adding examples: almost always worth trying, cheap, pick boundary cases, stop at three or four.
- Adding reasoning steps: worth trying on genuinely multi-step tasks, skip on classification and extraction, and never treat the trace as an audit trail.
- Both at once: change one at a time, or you will not know which one did anything. That is the next lesson.
Do this before moving on
Take a classification task with a handful of labelled examples you trust.
Run it three ways: bare instruction, instruction plus three boundary examples, and instruction plus a request to reason first. Score each on the same twenty items and note the latency.
The usual result: examples help clearly, reasoning does not, and reasoning costs noticeably more time. Seeing that on your own task is worth more than being told, and it inoculates you against the reflex to add reasoning to everything.
Go deeper
- Chain of thought is the reference for in-context learning and reasoning, including the faithfulness caveat above stated properly.
- Advanced prompting covers what to do when a single reasoning chain is not enough: sampling several and voting, searching with backtracking, and splitting into chained calls. Read it when you hit the ceiling of this lesson.
- Practice question: Explain in-context learning and chain-of-thought is the foundational version, and it is asked early in screens.
- Practice question: Chain-of-thought is not helping. What do you try next? is the senior version, and the answer this lesson prepares you for.
- Practice question: Is chain-of-thought faithful to the model's actual computation? is the trap above as a depth probe, and it separates people who have read about this from people who have relied on it.
Key takeaways
- Examples teach shape, edge handling and register. Boundary cases carry information; typical cases mostly do not.
- Three or four good examples beat ten similar ones, and every example is tokens on every request forever.
- Step-by-step reasoning helps when a careful person would need steps, and costs without helping on classification, extraction and lookup.
- Stated reasoning is generated text, not an audit trail. Do not show it as justification and do not debug by reading it.
Check yourself before an interviewer does. Answer from memory first.
You add a request to reason step by step to a sentiment classifier. Latency doubles and accuracy is unchanged. Why?