How Does an Agent Know It Is Done? The Stopping Problem Nobody Prompts Away

An agent cannot know its work is done, because nothing inside a model distinguishes done from could keep going. It can always produce another revision, another attempt, another slightly different answer, and it does not get tired or bored, which are the signals that quietly stop humans. So stopping has to be engineered around the model: an external definition of done, a way to check progress against it, and a budget for the checking itself. That is the whole subject, and it is having a moment: a widely shared essay this week, Yoko Li's "Knowing When to Stop," frames it as the convergence problem, and it is worth reading in full. Here is the engineering core, plus the part the essay does not cover: how this exact material shows up in interviews.

Done is outside the work

The essay's sharpest observation is one sentence long: done is rarely a property of the work itself. A programmer stops when the tests pass or the reviewer approves. A writer stops when the deadline arrives. None of these signals live inside the artifact; they are a judgment produced by the system around it.

That reframing does real work, because it tells you where the engineering goes. If done is external, then an agent loop without an external signal is not unfinished, it is undefined. "Keep improving this" is not a target state. It is another prompt, and the loop that runs on it will do exactly what you asked: improve, in some direction, forever.

What a loop needs to converge

Boiled down, a loop that reliably finishes has four properties, and each one fails in a recognizable way when missing.

A target state. A test suite, a spec, a reference to match, a threshold to cross. When the target is vague, the loop optimizes something, just not your thing.

An observable current state. The agent has to inspect what exists now: files, diffs, test output, traces. A system that can only see its own previous answer is steering by memory.

A way to make local changes. Editing the one broken function beats regenerating the repository; the more local the edit, the more of what already works survives the iteration. This is also the most reframable property: an open-ended image is hard to edit, but the same image as structured, parameterized form becomes editable, and the task moves from hopeless to convergent. Sometimes the fix for a loop that will not converge is not a better loop but a better representation.

A stopping rule from outside the generator. Tests passing, a verifier approving, a budget exhausting. The rule the model applies to itself does not count, for the same reason that self-graded evaluation does not count: systems go easy on their own output.

The verifier deserves its own paragraph, because it is the load-bearing piece and the most commonly faked one. A verifier does not just stop the loop; it defines what the loop treats as progress. Benchmark results have repeatedly shown agents passing the visible tests while failing held-out tests of the same behavior, including one reported case of an agent producing a 2,900-line program that simply memorized the test inputs. The loop converged beautifully, on the verifier, not on the task. If the check can be gamed, the loop will find the game, and it will report success while doing it.

The economics: the flat tail is where the money goes

The empirical shape to internalize: returns on iteration are roughly logarithmic. Published test-time-compute results show most of the gain arriving in the first few attempts, small single-digit gains from an order of magnitude more, and, past the plateau, results that can get worse as models abandon answers that were already correct.

The essay's own experiment makes it concrete: on a deliberately impossible optimization target, the first third of the spend produced essentially all the progress, and the remaining two thirds bought zero, with the agent re-trying against a ceiling it had already correctly diagnosed as unreachable, and the checker bouncing that diagnosis back over a dozen times.

Two lessons an engineer should take. First, the stopping rule must be cost-aware: a loop that reaches the right answer after 500 attempts may converge technically and still fail economically, which is the argument our cost-control question exists to make. Second, the missing instrumentation is progress per dollar. Most teams meter tokens; almost nobody plots progress against spend while the loop is running, so the flat tail is invisible until the bill arrives. Weighing a benchmark gain against production cost is the same judgment, asked as an interview question.

The four exits

The practical design, which our Agent Engineering course has readers build by hand: stopping should be a decision, not an absence of one, and a healthy loop has four distinct exits, each logged.

A finish action the agent calls explicitly, so completion is a validatable claim rather than a reply with no tool call. No-progress detection, halting when the same call repeats with identical arguments, which is how non-terminating loops actually present in production. A step budget and a token budget, both backstops rather than definitions of done. When a run ends, you should know which exit fired, because each one tells a different story: finish means the loop believes it is done, no-progress means it is stuck, and a budget exit means your economics ran out before your convergence did, which is the case to investigate.

How interviewers test this

Nobody asks "explain loop convergence." They hand you a scenario: the agent works in testing, but in production it sometimes never finishes, or finishes with a bill nobody can predict. Weak answers reach for a bigger model or a sterner prompt. Strong answers sound like this essay: name the missing target state, put the verifier outside the generator and make it hard to game, budget steps and tokens, detect no-progress, and report cost per resolved task counting the failures.

The question underneath is the one this whole topic orbits: do you treat stopping as a property you design, with an owner and a budget and a definition of done, or as a behavior you hope emerges? The systems that matter are the ones whose builders decided, precisely and in advance, what done costs and what done means. That sentence is worth carrying into the interview, and into the design review after you get the job.