What an AI agent does when it does not know the answer (and what yours does today)
A well-designed agent does three things, in this order: it detects that it does not know through a measurable signal, it abstains from answering or acting, and it hands the case off to a person with the context already gathered. A badly designed one does exactly one thing: it answers anyway. Fast, well written and confident — which is precisely what makes the failure expensive, because nobody looks at it twice.
Hence the thesis of this guide, which sounds like an exaggeration and is not: an agent that never hesitates is the most dangerous one you can run. Doubt you can see — "I do not have this, let me pass you to someone who does" — costs you one transfer. Doubt you cannot see — a plausible false answer, a field filled in by guesswork, a refund approved with no basis — costs you the error plus the time to find it, and that second number is almost always bigger. So an agent is not judged by its best case. It is judged by what it does in its worst one.
What follows is not about tuning the model. Pushing invention down at the answer level — grounding, citations, decent retrieval — is a separate job and you should do it: that is reducing AI hallucinations in production. This guide is about everything around the moment of doubt, which you design in your own system and which works the same for a support agent, one reading invoices and one drafting a report. The specific handoff criteria for a support chatbot are in building an AI chatbot; here is the general piece.
Your agent is trained never to stay quiet
Before designing anything you have to accept the starting point, and it is uncomfortable: the model underneath ships with a factory bias against silence. This is not a contrarian consultant opinion. Kalai, Nachum, Vempala and Zhang put it plainly: language models hallucinate because training and evaluation procedures reward guessing over acknowledging uncertainty, and the problem persists because of how most evaluations are graded — models are optimised to be good test-takers, and a good test-taker answers anyway when it does not know. Source: Why Language Models Hallucinate, Kalai, Nachum, Vempala and Zhang (OpenAI), 4 September 2025, accessed 13 September 2026.
The operational corollary is what matters: abstention is not a dormant behaviour you can wake up with one line in the prompt. "If you do not know, say so" helps at the margin and fails exactly where it hurts, because you are asking for caution from a system trained to do the opposite. Doubt has to be manufactured outside the model: signals the system computes, checks that run before any action, and a route to a human that actually exists.
There is a second trap, even more popular: asking the model how confident it is. It does not work, and it has been measured. Xiong and colleagues, evaluating confidence elicitation, found that models are highly overconfident when verbalising their confidence, with values overwhelmingly concentrated in the 80% to 100% band. Source: Can LLMs Express Their Uncertainty?, Xiong et al., ICLR 2024, accessed 13 September 2026. A number that almost always says ninety is not a signal. It is decoration.
The doubt signals that work, and the one everybody uses
Useful doubt does not come from the model. It comes from the system around it. These are computable signals, each with its own trigger, and none of them needs the model to be honest.
| Signal | Where it comes from | What triggers it |
|---|---|---|
| Source coverage | The retrieval step, not the model | No passage above threshold, or the one you got does not contain the fact being asked for |
| Disagreement with itself | Two passes over the same input | The two answers do not match on the fact that matters |
| Broken output contract | Validation, before anything gets written | A required field is missing, an amount is not a number, a date does not exist |
| Hard business rule | Your policy, not the model | Amount over the limit, customer with no history, regulated data, country out of scope |
| Failure counter | The orchestrator | N retries or N turns with no progress |
| Explicit request | The person on the other side | "I want to talk to someone" |
The first three can ship the same day and catch the most. Source coverage is the one with the highest return per euro: if the agent answers over your documents and retrieval brought back nothing relevant, you already know that whatever comes next is invention — without reading it. Binary, cheap, and it fires before the answer exists.
One nuance that avoids the most common design mistake: these signals do not add up into one magic number. Each one calls for a different treatment. Empty coverage is resolved by saying "I do not have this documented" and handing off. A broken contract gets one bounded retry and then, if it breaks again, the queue. The hard rule negotiates with nothing: if the amount crosses the limit, it goes to a person even if the agent is dead certain. A single global confidence score blends things that are not alike and ends up hiding the first three.
Verify before acting: doubting an answer is not doubting an action
Abstention has two versions and confusing them is expensive. Abstaining from answering is cheap: if you go quiet too often, someone answers later. Abstaining from acting is a different league, because a bad action leaves a trace in a system and undoing it costs more than doing it. The bar cannot be the same in both places.
OpenAI's guide to building agents organises this by tool, not by conversation: it recommends assessing the risk of each tool available to the agent and giving it a rating — low, medium or high — based on read-only versus write access, reversibility, required permissions and financial impact; and then using that rating to trigger automated actions, such as pausing for guardrail checks before executing high-risk functions or escalating to a human. On human intervention it is just as concrete: the two usual triggers are exceeding failure thresholds — limits on retries, or failing to understand user intent after multiple attempts — and high-risk actions that are sensitive, irreversible or high-stakes, such as cancelling orders, authorising large refunds or making payments. Source: A practical guide to building agents, OpenAI, accessed 13 September 2026.
- Check before you write. If the agent is about to touch a system, the fact is verified against the source of record — the master, the ERP, the contract — not against what the model remembers. On actions, verification is not an improvement. It is the step.
- If it is not reversible, no threshold applies. No confidence measure authorises an action you cannot undo. That one gets confirmed, always, even if the agent has gone six months without a miss.
- If the check cannot run, that is the doubt. The system that should confirm is down, the document will not open, the identifier does not exist. This is not an edge case to resolve by analogy. It is a case that stops.
This connects to two decisions that are made separately and are worth not mixing: what the agent may touch is in what permissions to give an AI agent, and how much it may do alone is in the autonomy levels of an agent. Doubt behaviour is the third leg: what happens when, with those permissions and on that rung, this particular case does not measure up.
Where to set the threshold without drowning your team
The threshold is not a number you pick off an article. It is the consequence of two things that are yours: the cost of the error and the capacity of the queue. If the error is cheap and visible, the threshold can be loose. If it is expensive and silent, you tighten it. And if the number of cases it sends to review does not fit in the hours your team has, that threshold is not prudent: in three weeks it will be approved in bulk without reading, which is worse than not having it, because it also produces a log claiming somebody looked.
- Group the agent's tasks into three buckets by cost of error: cheap and visible, expensive and visible, expensive and invisible. The third one rules: there the threshold goes wherever it needs to go, however much it hurts.
- Start deliberately pessimistic. It is easier to defend a threshold you loosen with data than one you tighten after an incident.
- Measure for two weeks and look at the abstentions, not the hits. For every case that went to the queue, mark whether the person resolved it differently — the abstention was right — or exactly as the agent would have — it was noise. That split is what moves the threshold, not a gut feeling.
- Adjust by segment, never globally: new customer versus ten-year customer, small amount versus large, a language you have documentation in versus one you do not.
- Make capacity a hard ceiling. If at the correct threshold the queue does not fit, the conclusion is not to lower the threshold. It is that this task should not be in production yet without more hands.
Once volume grows this stops being a configuration and becomes a function with shifts, coverage and its own metrics: that is human oversight of AI at scale, and it is best built before you need it.
What has to travel in the handoff so the person does not start from zero
A handoff without context is doubt well detected and badly resolved. The agent did its part — it stopped in time — and the customer still tells their problem a second time, which is exactly what makes a chatbot infuriating. In an internal handoff the same thing happens with nobody complaining: the person rebuilds by hand what the system already had.
The mechanics are solved in the two reference guides. In the decentralised pattern OpenAI describes, a handoff transfers control and also the latest conversation state; and an agent, by definition, can halt execution and transfer control back to the user when it fails. Anthropic frames it as stopping points: agents can pause for human input at checkpoints or when they hit a blocker, and it is worth setting stopping conditions — a maximum number of iterations — to keep control. Sources: A practical guide to building agents, OpenAI, and Building effective agents, Anthropic, 19 December 2024; both accessed 13 September 2026.
What the documentation does not give you is the contents of the package. These are the six fields that make a handoff save time instead of costing it:
- What the case was asking for, in one line, in the words of whoever asked.
- What the agent did before stopping: what it queried, what it found, what it wrote if it wrote anything.
- Why it stopped: the specific signal that fired, not "low confidence". "No document covers the question" and "the amount is over the limit" ask two different things of the person.
- What it has already gathered: identifiers, attachments, verified data. If the person has to ask for them again, the handoff failed.
- What it has NOT done, when it matters: "nothing was sent to the customer", "the order was not touched". This prevents the duplicated action, the classic escalation accident.
- What is expected from the person: decide, draft, authorise. A case that arrives with no verb sits in the queue.
And the return leg is missing, which almost nobody builds: the person's decision has to go back into the log with its reason. Without it, every abstention is a lost case; with it, the review queue becomes the case set you measure the agent against next month. That is the difference between a system that escalates and one that also learns from what it escalates.
How to measure an agent that doubts well (and why your eval punishes it today)
If your only metric is accuracy, you are reproducing in-house the exact problem Kalai and co. describe: when "I do not know" scores the same as a wrong answer, guessing is the rational move. They call it an epidemic of penalising uncertain responses, and their proposal is not to add one more hallucination eval but to change how the evals you already run are scored. In your company it is the same operation, and it fits in three numbers.
- Abstention rate. What share of cases the agent stops. There is no universally good value: there is an expected value you set and a trend you watch. If it drops on its own from one month to the next without you touching anything, something broke in the signal.
- Abstention precision. Of what it sent to the queue, how much the person resolved differently from how the agent would have. This is the metric that separates a cautious agent from a cowardly one.
- Confident error. Cases that went wrong with the agent flagging nothing. It is the only one of the three with a clear target, and the target is zero. Each one gets opened and turned into a new signal or a hard rule.
With those three numbers the conversation moves: it stops being "is the agent good?" and becomes "where does it get things wrong, and does it know?". Which is the only version of the question you can answer with data.
We build this behaviour before the agent: computed signals, mandatory verification on everything it writes, a threshold tied to real capacity and a handoff with the six fields. It is what sits underneath 24/7 AI support and under any agent we let talk to a customer of yours, and what gets watched at volume in human oversight of AI at scale. The rest of the build — permissions, autonomy, memory, evals — is in building an AI agent that survives production.