Illustrative scenario. A distributor launches an agent that sorts order issues and drafts the reply to the customer. Launch day comes with cake. By week three, three people have opinions on whether it's going well, and each is looking at something different: one counts the cases it closes, another the ones that come back, and the third only remembers the one that went wrong. Nobody has a date to decide whether the agent stays, gets tuned or gets switched off.
The thesis in one line: launch isn't the finish line, it's the start of the cheapest window you'll ever get to fix an agent, and that window gets wasted without a calendar. Each week of the first thirty days has to answer a different question, and at the end someone with a name has to sign a decision.
What happens in the first weeks of an AI agent in production?
In the first weeks of an AI agent in production, you get what the demo couldn't show: real cases arrive, with odd formats, exceptions and impatient customers. It's also when fixing things costs least. Volume is still low, changes ship without asking half the company for permission, and people still remember how the work was done before, so they spot quickly when the agent drifts.
After that, every fix gets more expensive: more volume, more people depending on the agent, more history you don't want to break. That's why the first month deserves its own rules rather than "the agent is live now". Four signals get checked every time: exceptions (what the agent can't resolve), human corrections (what people fix after it), drift (how far it moves from what you tested) and the real cost per case.
Week 1: does it do what we signed off?
Week one isn't measured, it's read. One person reviews a wide sample of cases by hand every day, ideally someone who helped build the agent and someone who knew the old process. Fifteen minutes a day is enough if the sample is well chosen: the easy cases, the doubtful ones and every case the agent escalated.
This week's question is whether the agent does exactly what was agreed, and only that. Three checks: it touches nothing outside its perimeter (the permissions you gave it, not the ones it could have; the autonomy levels of an agent work as a checklist), every case leaves enough trace to reconstruct what happened, and there's a list of "cases nobody saw coming" with a date and an owner. That list is the most valuable asset of the week: it's the real work the demo skipped.
Week 2: what do people fix after the agent?
In week two you look at what humans correct. Every correction is free information about where the agent fails, but only if you record what was corrected and why. The practical rule is simple: one correction is a mistake; the same correction three times is a missing rule.
Sort corrections by type: wrong extracted data, wrong decision, tone or format, and a case that should have been escalated and wasn't. The last one costs the most, because it means the agent doesn't know when it doesn't know; the criteria are in what an agent does when it doesn't know. And compute a performance metric with the same yardstick you'll use later, as the guide on measuring the performance of your automations explains: change the metric mid-month and you can't compare anything.
Week 3: are exceptions piling up and is the ground moving?
Week three is about exceptions and drift. Exceptions are the work the agent hands back to people, and what matters isn't how many there are but whether anyone handles them, how late, and whether the queue grows or settles. A queue nobody prioritised is where savings go to die, as the exception queue, where savings die describes.
Drift is measured against something concrete: a bank of cases with known correct answers, the same one you used to approve the launch. Run it again now and check whether the agent still answers the same way. If the model provider, an input format or a connected tool changed, you see it here before the customer does. Behaviour getting worse over time isn't bad luck, it's the norm, and we explain it in why your pilot worked in January and not in June.
Week 4: what does each case really cost, and what do we decide?
Week four puts a price on it. Cost per case isn't the model bill: it adds the hours of whoever reviews, whoever handles exceptions and whoever retunes, plus the time spent building the report. If the only cost on your screen is tokens, you're looking at the cheap part. To get those numbers live without hand-building the dashboard, there's the function of monitoring AI in production.
With the four weeks in front of you, a decision gets made in writing. There is no fourth option called "we'll see":
| Decision | When it applies | What it means |
|---|---|---|
| Declare stable | Repeated corrections drop, the exception queue isn't growing and cost per case is within what was agreed | Moves to sampled review and the maintenance routine |
| Extend with changes | There's an identified, fixable cause, but the numbers don't yet support the case | Set a deadline and one concrete change, not "a few more days" |
| Stop or reduce autonomy | The same corrections show up every week or nobody can explain the cost | Back to assistant mode or switched off; a valid outcome, not a failure |
When should you NOT declare an agent stable at the end of the month?
There are four situations where signing "stable" is a mistake even if the numbers look good. One: volume was so low over the four weeks that no odd case showed up; a quiet month proves nothing. Two: the month included a provider or process change and the case bank wasn't re-run afterwards. Three: whoever built the agent made the call without the person accountable for the process signing; that's the subject of acceptance criteria for an AI agent: who signs off. And four: nobody has a name and a shift assigned to watch the agent next week.
Declaring stable doesn't mean you stop looking. It means moving from intensive review to a lighter routine, with the schedule for maintaining AI automations and the agent already in your inventory. If the jump from pilot to operation is exactly where you're stuck, the function of taking an AI pilot to production covers that stretch.
What you take away
What this calendar changes isn't the technique, it's the habit: one question a week, one named person and a decision at the end. Week one, does it do what was signed? Week two, what do people correct? Week three, are exceptions piling up or is the ground moving? Week four, what does it cost and what do we decide? An agent that survives those four questions has earned a boring life, which is exactly what you want from it.
Note: this article uses no third-party figures. The scenario is illustrative, and the deadlines and thresholds for each decision depend on each company's process, volume and risk; set them before launch, not after.