Why an automation breaks (building is not the same as maintaining)
An AI automation is not a piece of furniture: you don't build it, plug it in and forget it. It's an engine. And an engine that works every day wears down, needs oil, and one day stops if nobody looks at it. The promise of "we leave it running and you never touch it again" sounds great in the sales meeting. In production, it's the first thing that falls apart.
The important detail is how it breaks. Almost never with a bang. An automation rarely goes down all at once —that at least you'd see—: it degrades quietly. It keeps working, just a bit worse each week, until one day someone checks and finds it's been classifying wrong, answering off-tone or skipping cases for a month. By then the damage is done. Maintaining is the opposite: looking before anyone complains.
This isn't a flaw of your particular automation, it's how they all work. In fact, leaving maintenance out of the equation is one of the reasons automation projects fail: the launch gets celebrated, nobody owns the care, and six months later the system is a zombie no one dares switch off or fix. If you want the wide map of all this, it's in the guide to automating with AI; this is the zoom into the part that starts the day after delivery.
The three things that break: the model changes, the integration snaps, the odd case shows up
An AI automation stands on three legs, and all three move over time. Knowing which they are is knowing what to watch.
One: the model changes. The provider updates the version, tweaks the behavior, deprecates the one you used. It doesn't warn you to your face: one day the same prompt returns an answer with a different tone, a different format or less accuracy. The system doesn't stop —that's the problem—, it just starts deciding a bit worse. It's the sneakiest break because no alarm goes off on its own, which is why surviving a model version change without breaking your automations is something you prepare for beforehand, not on the day it happens.
Two: the integration snaps. Your CRM changes a field, an API updates its format, a permission or a token expires, someone touches an export. This one you see fast: the flow stops dead. It's the hard break, the one that shows its face. And it's why integrating AI with your systems is not a one-off job: every system you hook into is a part that can move on its own.
Three: the odd case of month 3. At first the automation sees the normal cases it was designed for. But the real world has a long tail: the order with the strange condition, the email in a language you didn't expect, the customer who fills the form backwards. That case didn't exist when you built it; it shows up weeks later and breaks a flow that looked solid. No automation is born covering 100% of reality; it covers it as reality teaches it.
| What breaks | How it shows | How you catch it |
|---|---|---|
| The model changes | Still answers, but worse: other tone, other format, less accuracy | Log and sample review; no alarm on its own |
| The integration snaps | The flow stops dead | Error alerts; seen the same day |
| The odd case shows up | A new type of input jams or diverts the flow | Exception queue and review of what escalated |
What maintenance actually is (you care for an engine, you don't hang a painting)
Maintaining isn't "being around just in case". It's concrete work with concrete tasks. You care for the engine: review how it decides, adjust what drifts, patch the holes real use reveals. You don't hang a painting and admire it from afar.
In practice, maintaining an AI automation is four things:
- Supervise the decisions. Look in the logs at what the system is deciding and compare it against what a person would decide. Where it gets it right, it stays; where it fails, you fix it.
- Iterate the prompts and rules. Adjust the instructions when the model changes or when a new pattern appears. This is not reprogramming: it is tuning.
- Fix the edge cases. Collect the exceptions that escalated, understand why, and decide whether to cover them or keep routing them to a human. This is where the human-in-the-loop boundary lives, and it is not fixed: it moves with what the data teaches.
- Watch the integrations. Check that the connections to your systems are still alive and act when one goes down, before the whole flow stops.
All of this has a cost, and putting it in from the start is the honest thing. It's one of the three legs of the profitability math: setup, model, and maintenance. If you do the sums and drop the third, the number lies. That's why the ROI of automation guide insists on adding the maintenance hours before you sign: a system that's cheap to build can be expensive to keep.
How often you review it and what you look at each pass
Maintenance isn't staring at the system all the time nor never looking at it. It's a cadence. Some things are watched automatically and trip on their own, and some need a pair of eyes every so often.
- Continuous (automatic). Error alerts: if an integration goes down or a flow stops, it trips instantly. This does not wait for the review; it is the net that flags the hard break.
- Weekly (a quick pass). A glance at the logs and the exception queue: what escalated, what new case types appeared, whether the error volume is rising. Ten minutes that stop a small problem from becoming a lost month.
- Monthly (quality review). A sample of the system's decisions compared against human judgment, to catch the quiet drift of the model. This is where you notice if the provider changed something underneath.
- When the provider updates the model. Off-cadence: if a new version ships or yours gets deprecated, you test it before letting it run alone. A model change is treated as a part swap, not a detail.
The rule underneath it all: what breaks at once is watched automatically; what degrades slowly is reviewed by hand. You need both. Alerts alone won't catch model drift; manual reviews alone leave you finding out too late about a downed integration.
How you leave it with a safety net: logs, alerts and an owner
An automation that can't be maintained is an automation that isn't finished. And to be able to maintain it you need three things from day one, not as a patch afterwards.
Logs, to see what the system decided and why —without a record there's no supervision, only faith—. Alerts, so the hard break finds you before it finds your customer. And an owner: a named person who takes care of reading the logs, handling the alerts and iterating. Without an owner, nobody does the maintenance, which is the same as not having it. Who that owner is and how much autonomy they have is part of your team truly adopting AI, not just launching it.
And a boundary worth saying out loud: maintenance decides what gets reviewed every month, not what happens the night something stops. That other half —who gets called, how fast, and what the business does in the meantime— is who responds when an automation goes down: severities, on-call and runbooks at small-company scale.