Illustrative scenario. The vendor shows the support agent: it answers six questions well and the room applauds. The milestone is invoiced. Three weeks later the head of customer service says it’s “not finished” because it gets lost on returned orders; the vendor replies that was never in scope. Neither is lying. Nobody wrote down what “finished” meant.
The thesis in one line: an AI agent isn’t delivered when it works in the demo, it’s delivered when it meets acceptance criteria written before the start and signed by someone in the business. Without that piece of paper, the project either never ends or ends out of exhaustion.
What are AI agent acceptance criteria, and why isn’t a demo enough?
AI agent acceptance criteria are the measurable conditions the agent must meet, on real cases and over an agreed period, for the client to accept it as delivered. On a web form, “it works” is binary: it saves the data or it doesn’t. An agent answers differently every time and is right some proportion of the time, so “it works” is always an incomplete question: how often, on which cases, and at what cost when it fails?
A demo answers none of the three, because whoever is selling picks it: the best cases on the best day (we unpack this in the demo used your best cases). And the root problem isn’t technical. RAND’s analysis of why AI projects fail, based on interviews with engineers and data scientists, lists among the most common causes that the problem to be solved was misunderstood or miscommunicated (RAND, 2024). Acceptance criteria are that conversation put in writing, with a date and names.
What should an AI agent acceptance document include?
Six things, and all six fit on one page. If one doesn’t fit, you don’t yet know what you’re buying.
- Test cases that come from your operation, not the vendor’s. A closed set of real cases, with the hard ones and the exceptions inside, frozen before you start. If the set changes mid-project, results aren’t comparable. How to evaluate: evaluating the quality of AI agents and evals: without measuring, you don’t know if it works.
- Tolerated error, by type of error. Not a global “95% accuracy”: getting the tone wrong doesn’t cost what promising a refund the policy doesn’t allow costs. Why the average misleads: 95% accuracy isn’t enough in production.
- What it does when it doesn’t know. Escalation criteria: which situations go to a person, with what context and how fast. The design is in what an agent does when it doesn’t know.
- What it can touch, and how far. Which systems it reads, which it writes to and which actions need human approval, following an agent’s autonomy levels.
- How it’s measured, and for how long. A supervised trial period in real production, with the metric defined up front. What to measure without fooling yourself: measuring the performance of your automations.
- Who signs. A name and a job title from the business, not a department.
Who has to sign off that the agent is finished?
The person who bears the consequences if it fails: the process owner. Not whoever buys the technology, and not whoever builds it. The vendor proposes the criteria and shows they’re met; the business decides whether they’re good enough. If only the technology team signs, what’s being accepted is that the system is up, not that the work gets done well.
The signature includes the right to reject. An acceptance document you can only sign is paperwork. And a second signature helps: whoever will maintain the agent afterwards. Accepting something nobody knows how to run just pushes the problem to next month (see who maintains AI agents once they’re in production).
| What you write before starting | What happens if you don’t | Who defines it |
|---|---|---|
| A set of real cases | The vendor tests with what it knows works | The business supplies, the vendor helps cover gaps |
| Tolerated error by type | A pretty average hides the expensive error | The business, based on what each failure costs |
| When it escalates to a person | The agent improvises exactly where it shouldn’t | Business and vendor, together |
| Permissions and actions needing approval | You find the real scope on the day of the incident | Business and security lead |
| Trial period and metric | “Finished” gets measured by gut feeling | The business, with the vendor as advisor |
| Signature | The project never ends, or ends out of exhaustion | A business role with a named person |
What happens when there are no acceptance criteria?
There are two possible endings and neither is good. In the first, the project never ends: every new request feels like a failure of the agent, the vendor feels scope growing without limit and the client feels they’re paying to fix what they already bought. In the second, it ends out of exhaustion: it’s declared good because nobody can face one more meeting, and the agent goes live without anyone knowing its real error rate.
The second ending is worse, because it looks like a success. It’s the contractual version of what we explain in why AI automation projects fail: the project doesn’t fail in the demo, it fails in the gap between what each side took for granted.
How do you write a criterion you can actually check?
With three ingredients: a set of cases, a threshold and a failure condition that isn’t negotiable. An example wording (the numbers are illustrative; you set yours based on what each error costs you):
Notice the asymmetry: the threshold is negotiated, the failure condition isn’t. Some errors are tolerated in a proportion and some aren’t tolerated at all. Writing that down avoids the most expensive argument in the project, the one that happens when the failure has already reached a customer.
When don’t you need a formal acceptance document?
When it’s a prototype declared as such, an internal experiment with no customers or personal data, or a case where being wrong is cheap and reversible. There, one sentence and a date will do: “this is a trial, reviewed on day X.” As soon as the agent touches customers, money or personal data, the page gets written.
If you already have an agent running and nobody remembers what was accepted, now is the time to write the document, before the next expansion. And if you’re about to hire one, evaluating the quality of AI agents is the work that turns this page into a set of cases and a threshold you can actually run.
The line to take away: an agent isn’t finished when it impresses, it’s finished when someone in the business, with a name and a title, signs that it meets what was written before starting.
Sources (accessed October 27, 2026): “The Root Causes of Failure for Artificial Intelligence Projects and How They Can Succeed,” RAND Corporation (2024). The opening scenario and the example figures are illustrative, not a real case.