"It works great." It's the most-said thing about an AI system and the least defensible. Someone tried it for a while, asked it five questions, they came out fine, and from there to "it works" is a leap of faith that production makes you pay for. How to evaluate an AI system in the enterprise isn't that impression: it's measuring, against a baseline, whether it's right —and how much it's wrong when it is.
The thesis in one line: without an evaluation set —what the trade calls evals— you don't know if your AI works, it just looks like it does. "It looks like it works" isn't a metric; it's an anecdote with good luck. What separates a system you can put in front of a customer from an experiment praying not to fail is, almost always, whether there are evals or not.
How to evaluate an AI system in the enterprise: why "it looks like it works" isn't a metric
Evaluating an AI system in the enterprise is comparing what it answers against what it should answer, over a set of representative cases, and getting a number. Concretely: you gather a set of real inputs —the questions, documents or tasks it actually gets—, define the correct answer or the pass criterion for each, and run that set every time you change something —the prompt, the model, the data. The result is a score that moves: it goes up when you improve, down when you break something without noticing. Without that number, every change is blind and every "it works" is an opinion.
What an eval is (and how it differs from "trying it for a while")
Testing a system by hand is useful on day one and misleading on day two. An eval is the opposite of that informal test in three ways:
- It's representative, not anecdotal. Not the five questions you happen to think of; dozens or hundreds of real cases, including the odd ones and the ones that already failed once.
- It's repeatable, not one-off. It runs the same way on every change, so it compares apples to apples: you know whether today's version is better or worse than yesterday's, not whether it "felt good".
- It gives a number, not an impression. A hit rate, an error rate, a hallucination rate. Something you can put on a chart and watch every week.
The difference between a serious system and a pretty demo is, almost always, whether that number exists. The technical how of building it —the set, the criteria, the loop— we break down in the guide on how to train an AI agent: data, evals and governance; here what matters is why without it you're flying blind.
What to measure: the four things that separate production from faith
You don't need to measure everything from minute one. Four metrics tell you more about your system than most projects that have been running for months:
- Hit rate. Of the cases in the set, how many it answers correctly by your criterion. It's the mother metric; if you don't have it, you have nothing.
- Error and hallucination rate. How much it's right isn't enough: what matters is how much it's wrong with confidence. A system that's right 90% of the time but invents the other 10% with total assurance can be worse than a humbler one.
- Edge-case coverage. What percentage of the odd cases —the ones that break weak systems— go through the eval. What's not in the set isn't measured, and what isn't measured degrades silently.
- Drift over time. The same eval, run every week. An AI system doesn't stay still: the input changes, prompts degrade, the provider updates the model. Drift shows up on the chart before it shows up in the customer complaint.
The mistake of not having evals from day one
The most expensive failure isn't having bad evals: it's having none. A system with no measurement doesn't fail on launch day —it fails slowly, and nobody notices until the error reaches the customer. Without a baseline number you can't prove it's improving, you can't defend the spend to leadership, and you can't tell a change that helps from one that breaks. It's, in fact, one of the named causes AI automation projects fail: zero measurement, zero way to know whether the project is still alive or a zombie nobody dares switch off.
Evals isn't a project, it's a habit
The trap is treating evals as a phase ("we'll build them at the end"). At the end is too late: the system is already in production making decisions you can't audit. Cheap and sensible is the other way round —start with a small set on day one and grow it with every odd case that shows up. It's not a deliverable you sign and forget; it's the routine that keeps the system honest.
Measuring well isn't glamorous, but it's the only thing that turns "we think it works" into "we know it works". If you're going to put AI to work in your company and want it to hold up, this is foundation, not decoration —and it's exactly what we build in enterprise AI infrastructure: the system, its evals and the loop that keeps it measured over time. We don't leave you an AI that looks like it works; we leave you one you can prove works —or one where you know exactly where it fails.