Model drift
maintenancedrift · model decay
The quiet decay of an AI system that used to get it right: the world shifts, the inputs shift, or the model underneath shifts, and quality drops without a single error being raised.
In classic ML, drift came from the world: production data stopped looking like training data and the model aged. With language models there is a second source that did not exist before: the vendor retires or updates the model you depend on, on their schedule and not yours. Of the two failures, the loud one is cheap: the model shuts down, the API throws, the logs shout. The expensive one is silent: you migrate to the successor, the call still returns 200, and behaviour moves underneath you. Which is why the defence is not uptime monitoring. It is a suite of cases with expected outcomes run before and after every change, measuring the distribution of what goes in as well as what comes out, and treating the vendor's published retirement date for what it is: a maintenance date already on your team's calendar.
How it differs from
- Hallucination
- A hallucination is one wrong answer. Drift is a sustained drop in quality you can only see in aggregate.
- Outage
- An outage shows up on the dashboard in minutes. Drift breaks nothing: it returns worse results with the same status code.
Examples
- A ticket classifier built on last year's catalogue: the catalogue changed, the classifier did not.
- The vendor announces a model retirement months ahead. The date is public; the migration still will not do itself.
FAQ
- How do you detect drift when nothing errors?
- By measuring on a schedule, not by waiting for alerts. A suite of cases with expected outcomes run on a cadence and on every change; business metrics per cohort (resolution rate, escalations, retries); and input monitoring, because very often what changed is not the model but what reaches it.
- How often should a production agent be reviewed?
- At minimum on every model, prompt or data-source change, with a fixed cadence in between. The vendor's deprecation calendar sets the floor: if your agent will outlive the model version that drives it, migration is part of the cost of running it.