Agent evaluations (evals)
Practicaevals · agent evaluation
Agent evaluations (evals) are systematic, repeatable tests that measure whether an agent does its job well: accuracy, safety, cost and behavior on hard cases. They are to an agent what tests are to software: without them you cannot tell whether a change improved it or broke it.
An eval is a set of cases with an expected answer (or a judging criterion) that the agent is run against again and again. They serve three purposes: deciding whether an agent is ready for production, catching regressions when you change a prompt, a model or a tool, and monitoring live that quality does not degrade. Without evals, 'the agent is doing better' is an opinion; with evals, it is a number. It is the foundation on which you can grant more autonomy sensibly.
How it differs from
- Monitoring
- Monitoring watches the agent live; evals test it against controlled cases, including before production.
Examples
- Running 200 typical conversations with an expected answer every time you change the agent's model is an eval.
- Measuring the share of correct resolutions and the average cost per case before shipping an agent to production.
FAQ
- How often do you run evals?
- On every relevant change (prompt, model, tool) and periodically in production. The point is that no change reaches the customer without passing the battery of cases.