Why the same prompt gives a different answer twice
Randomness in a model is a setting, not a mystery, and knowing which knob does what is the difference between a feature that behaves and one that surprises you in production. This lesson also covers why turning it all the way down still does not give you exact repeatability.
TL;DR: A model does not pick the next token, it produces a probability distribution over all of them, and a separate step chooses one. That choosing step is yours to control. Turn it down for anything a program consumes, leave it up for anything a person reads, and do not expect bit-identical output even at zero, for reasons that have nothing to do with the setting.
Where you are. You know what the model computes. The last step before you see text is one setting, and it is the one a beginner is most likely to leave at its default and then be confused by. It is also the shortest lesson in the module.
The step nobody mentions
At every position, the model outputs a score for every token in its vocabulary, converted into probabilities. Something like: "blue" 61%, "grey" 14%, "clear" 9%, "dark" 3%, and a long tail of everything else.
The model's job ends there. Choosing one of those tokens is a separate step, and it is configuration rather than intelligence.
That is the whole idea. Two runs of the same prompt differ not because the model changed its mind but because the choosing step made a different draw from the same distribution.
The two knobs worth knowing
Beginner material lists five. Two of them carry nearly all the effect, and the rest are refinements.
Temperature reshapes the distribution before choosing. Low temperature exaggerates the gap between likely and unlikely, so the top candidate dominates and output becomes repeatable. High temperature flattens the differences, so unlikely tokens get a real chance and output becomes more varied and less predictable. At zero it always takes the most probable token.
Top-p truncates instead of reshaping. Sort the candidates by probability, keep adding them until their combined probability reaches your threshold, then sample only from that set. The point is that the set resizes itself with the model's confidence: where the model is sure, it may contain one token; where it is genuinely uncertain, it may contain dozens. That adaptiveness is why it tends to behave better than a fixed cutoff.
They interact, which is why changing both at once makes debugging hard. Move one.
The setting to use
There is a straightforward rule, and it holds far more often than it fails:
If a program consumes the output, drive randomness to the floor. If a person reads it, leave some in.
Classification, extraction, routing, anything parsed as JSON, anything that must match a fixed set of values: you want the same input to produce the same output, and variety is pure downside. Drafting, brainstorming, conversational replies, anything where the same phrasing every time would feel mechanical: variety is the feature.
The failure mode to recognise: a team ships an extraction feature at the default setting, sees occasional malformed output, and adds retries and parsing patches. The actual fix was one number.
Why zero is not the same as repeatable
This is the part that surprises engineers, and it is worth knowing before it bites.
Even with randomness at zero you can send the same prompt twice and get different text. Nothing is broken. Providers batch requests from many users together, and the arithmetic underneath is floating point, where changing the grouping changes the last digits of a result. Usually that is invisible. Occasionally two candidate tokens are so close that a tiny difference flips which one is on top, and from that token on the outputs diverge and stay diverged.
So the honest framing: zero temperature gives you the most likely answer, not a guaranteed identical one. If your system genuinely requires identical output, get it from caching the result, not from a setting.
Do this before moving on
Take one prompt with a factual, short answer. Send it five times at a high temperature, then five times at zero, and keep both sets side by side.
Two things to look for. How much the high-temperature answers differ, which tells you what your users would see at defaults. And whether the zero-temperature runs are truly identical, which tells you whether the point above is live on the model you are using. Then decide what your own feature should be set to, and write the reason down.
Go deeper
- Temperature and sampling is the reference: every knob, what each does to the distribution, and how to match settings to a task. Read it when you need the full set.
- Constrained decoding is the stronger tool for anything that must be valid JSON or a fixed set of values, and it beats lowering temperature and hoping. It matters in module 3.
- Practice question: How do temperature and top-p differ, and when do you change each? is a common early screen, and shallow answers are obvious.
- Practice question: Why is an LLM still non-deterministic at temperature zero? is the batching and floating-point point above, asked as a depth probe. Worth owning.
- Practice question: Which decoding strategy would you pick for this task? is where this becomes a design decision rather than a setting.
Key takeaways
- The model produces a distribution; a separate, configurable step picks a token from it. Randomness is a setting, not a property of the model.
- Temperature reshapes the distribution, top-p truncates it adaptively. Change one at a time.
- Program consumes it, turn randomness down. Person reads it, leave some in.
- Zero gives the most likely answer, not a guaranteed identical one. For true repeatability, cache.
Check yourself before an interviewer does. Answer from memory first.
An extraction feature returns malformed JSON on maybe one request in twenty. What is the first thing to check?