Thinking between actions, and what it costs
Interleaving a short reasoning step before each action is the most widely used agent pattern. This lesson covers what it actually buys, the three places it helps most, and the specific way it degrades over a long run.
TL;DR: Letting the model state a short rationale before each tool call improves tool choice and gives you a readable trace, at the cost of output tokens on every step and a context that fills with the agent's own commentary. Keep the reasoning short, keep it out of the permanent history where you can, and never treat it as an explanation of what actually happened.
Where you are. You know the loop accumulates. Now the first variation on it, and the most common: reasoning and acting interleaved, one short thought before each action. It is a default worth understanding rather than adopting unexamined.
The pattern
Instead of the model emitting a bare tool call, each step produces a brief rationale and then the call.
The user is asking about a refund on a specific order, so I need the order details first.
lookup_order(order_id="4471")
Then the result comes back and the next step does the same.
Two things change relative to a bare loop.
Tool choice improves, most noticeably when several tools are plausible. Stating the goal of this step before choosing narrows the choice, in roughly the way it does for a person.
You get a trace you can read. When a run goes wrong, "I could not find the order so I will search by customer email instead" tells you what the agent believed at that step. Debugging a bare tool-call sequence is considerably worse.
Where it earns its cost
Not everywhere. Three cases carry most of the benefit:
Many similar tools. Where selection is the hard part, stating the intent before choosing helps most. If you have three tools, less so.
Recovering from failure. After an error, a rationale step is where the agent decides to try something different rather than retry the same thing. This is where it most directly prevents the ping-pong failure.
Multi-hop tasks. When step three depends on interpreting the result of step two, the interpretation is worth making explicit.
Where it does not earn its cost: a two-step task with one obvious tool. The rationale is a paragraph of throat-clearing, billed and waited for on every request.
What it costs
Output tokens on every step, which is latency, because output is generated serially. A long rationale is a directly felt delay.
Context, permanently. The rationale lands in the message list and is resent on every subsequent step. By step eight you are paying repeatedly for step two's thinking. This is the accumulation problem from the previous lesson, arriving through the pattern that is supposed to help.
Confident narration of a bad decision. The rationale always sounds reasonable, including when the choice is wrong, which makes a broken run read as a competent one.
The trap worth naming
The rationale reads as an explanation of why the agent did something. It is not. It is text generated alongside the action, and it is a plausible narration rather than a record of a decision procedure.
Two practical consequences, both of which people get wrong.
Do not show it to users as justification. It looks like one and it is not verified to be one.
Do not debug primarily by reading it. If the agent chose the wrong tool, its rationale will explain confidently why that tool was right. Debug against what was called with what arguments, and what came back. Read the rationale for orientation, and trust the trace.
Keeping it from eating the run
Three cheap moves, in the order worth applying them.
Cap the length. "One sentence before each action" is a real instruction that works and saves noticeably at every step.
Drop old rationales from the history. Once a step has produced its result, its rationale has done its job. Keeping the result and the fact of what was tried, while discarding the commentary, keeps the useful signal and removes most of the volume. This is the highest-return trim available.
Skip it where it is not earning. If your agent has three tools and a predictable path, a bare loop is cheaper and no worse.
Do this before moving on
Take an agent with reasoning enabled and run a task that requires four or five steps. Print the message list at the end and mark each entry as rationale, tool call, or result.
Then work out what fraction of the tokens are rationales from steps that have already completed. It is usually a substantial share, and every one of them was resent on every subsequent step. That number is the argument for the trimming move above, measured on your own agent.
Go deeper
- Agent design patterns covers this pattern alongside the others and names them as you will meet them.
- Chain of thought is the underlying mechanism and includes the faithfulness caveat argued properly.
- Practice question: Is chain-of-thought faithful to the model's actual computation? is the trap above as a depth probe.
- Practice question: How would you design agent reflection or self-critique? is the same idea pushed further, where the agent evaluates its own output.
- Practice question: How do you trace and debug an agent in production? is what to rely on instead of the rationale.
Key takeaways
- A short rationale before each action improves tool choice and gives a readable trace.
- It earns its cost with many similar tools, after failures, and on multi-hop tasks. It does not on short predictable paths.
- Rationales accumulate and are resent every step, so old ones are the cheapest thing to trim.
- The rationale is generated narration, not a record of a decision. Do not show it as justification or debug by reading it.
Check yourself before an interviewer does. Answer from memory first.
Where does interleaved reasoning most directly prevent a known failure?