Illustrative scenario. A customer asks the support agent twice in one morning whether they can return an order after 40 days. At 9:10 the agent says: “Yes, you have 60 days.” At 11:30, same words, it says: “According to your order, the window is 30 days; contact an advisor.” Both answers sound reasonable, the same system generated both, and one of them is about to become a complaint. Nobody touched anything in between.
The thesis in one line: a model's variability isn't a bug you fix, it's a property you bound. You decide which parts of the process can never vary, which tolerate variation, and how you measure it before a customer sees it.
Why does AI answer the same question differently?
Because a language model doesn't look up a stored answer: it computes one every time. At each step it spreads probabilities across the possible words and picks one, and that pick includes a deliberate element of chance; the setting that controls it is called temperature. So the same question can produce different wording and, when the question leaves gaps, different conclusions.
Three more sources of variation tend to get forgotten. The first is what surrounds the question: the order in which the system retrieves documents, the date, the history or the memory all change the context and, with it, the answer. The second is the server: even at temperature zero, most inference services don't guarantee identical results, because the numerical computation depends on the load the server has at that moment (Thinking Machines, 2025). The third is the provider: a model version change or a settings change shifts behavior without you touching anything, the same effect we describe in why an AI agent gets worse over time when nobody has changed anything.
Can you remove variability by lowering the temperature?
Partly, and you wouldn't want to fully. Lowering the temperature reduces the spread in wording but doesn't eliminate it (the source above explains why), and it changes neither what the model knows nor what reaches it. It doesn't fix errors either: at low temperature a wrong model is wrong more consistently. We go deeper in reducing AI hallucinations in production. And some variation is exactly what you want: it's what lets a summary or a reply adapt to the case instead of sounding like a template.
The useful question isn't “how do I make it always answer the same?” but “where in the process can it not answer differently?”
Which parts of a process must be deterministic and which tolerate variation?
The rule: whatever the customer can quote, and whatever moves money or commitments, doesn't come from free generation. It comes from a rule, a data point or a template; the model, at most, words it.
| Part of the process | Tolerates variation? | Where it should live |
|---|---|---|
| Deadlines, amounts, terms, dates | No | In data from your system or a code rule; the model quotes it, it doesn't calculate it |
| Routing or classification decision | Little | Closed output (a list of categories), a confidence threshold and review of doubtful cases |
| Wording, tone, summaries | Yes | In instructions and examples; the model has room within them |
| Actions with effects (send, charge, delete) | No | In a deterministic check before executing, and tightly scoped permissions |
If almost all of your process falls in the “No” rows, you may not need an agent: a classic workflow is cheaper and easier to audit, as we explain in how to automate processes with agents, not chatbots. And what you do leave to the model is defined in its instructions: what it must cite, what it can't claim and what it does when it doesn't have the data.
How do you measure an agent's variability before it goes to production?
With a boring, repeatable test. It's a method, not a magic number:
- Gather real cases. A set of inputs from your operation, ambiguous and odd ones included, not the demo's.
- Repeat them. Run each input through the whole system several times (with its document retrieval, not just the model) and keep every output.
- Compare what matters. On decision fields (category, amount, yes or no), demand an exact match. On text, compare against a rubric: does it say the same thing in substance?
- Set a threshold per field. A field that moves money tolerates zero discrepancies; an internal summary tolerates more. Decide the thresholds before looking at the results.
- Rerun the test on every change. New model, new instructions, new document source: run it again.
That test is an eval, and designing and maintaining it is part of evaluating the quality of AI agents. Once the agent is live, the next step is measuring the performance of your automations, so variation doesn't slip in overnight.
What do you do if two contradictory answers have already reached a customer?
First, reconstruct what happened: without a record of the input, the retrieved context, the model version and the output, you can't know which of the two was right. Then close the gap:
- Take the data out of free text. If the contradiction was a deadline or a term, stop generating it: have it read from a single source.
- Make the system cross-check itself. For risky answers, generate twice and, if they don't match in substance, don't send: route to a person. It costs more per answer, so reserve it for where a mistake is expensive.
- Match autonomy to risk. What can't vary and you can't validate automatically gets reviewed by a human before it goes out. That's what the autonomy levels of an agent and the human in the loop are for.
When isn't it worth chasing consistency?
When the output is a draft a person will read and edit, when a mistake is cheap and reversible, or when variation is the value (ideas, text variants). Demanding zero variation there makes the system more expensive without protecting anyone. Spend the rigor where a different answer costs money or trust, and let the rest breathe.
The line to take away: don't ask the model to always be the same; design the process so that what can't change doesn't depend on it.
Sources (accessed October 29, 2026): “Defeating Nondeterminism in LLM Inference,” Thinking Machines Lab (September 10, 2025), used qualitatively. The opening scenario is illustrative.