Testing AI systems: how we know it works before you rely on it
A model that is right most of the time cannot be tested like ordinary software. The evaluation set, the metrics, and the bar before launch.
The short answer
Ordinary software either does what the test says or it does not. An AI system is right most of the time, and the question is how often, on which cases, and how badly it fails when it does. So it is tested statistically: a set of real cases from your own work, each with a known good outcome, scored before launch and before every change. The metrics are the ones the business cares about: accuracy on your cases, the rate of confident wrong answers, correct refusals, cost and speed. An adversarial pass adds the cases where someone tries to break it. And the bar is set before building, so that a system which does not clear it does not launch, however good the demo looked.
The evaluation set
| Element | What it is | Why |
|---|---|---|
| Real inputs | Fifty to a few hundred actual cases from your work, covering the common and the awkward | Synthetic examples miss what real ones contain |
| Labelled outcomes | For each, what good looks like and what is unacceptable, decided by the people who do the work | The system is measured against expert judgement |
| Edge cases | The unusual, the ambiguous, the ones that went wrong last year | Failure lives at the edges |
| Adversarial cases | Attempts to inject instructions, extract data, push off topic, provoke commitments | Production includes people who are not cooperating |
| Fresh additions | New cases from production every month | The distribution drifts; the set must follow |
The metrics that matter
- Accuracy on your cases, by category, not one blended number.
- Confident wrong answers: rate and examples. This is the metric to watch hardest.
- Correct refusals: does it decline what it should, and not what it should not.
- Grounding: are factual statements traceable to the sources provided.
- Guardrail behaviour: did the adversarial cases get caught.
- Cost and latency per case at realistic input sizes.
- Human-in-the-loop load: how many cases would reach a person, and were those the right ones.
Setting the bar
Before building, agree what launch requires: accuracy above a threshold on each category, confident wrong answers below a small rate, all adversarial cases caught, cost within budget at projected volume. Write it down. When the evaluation is run, the numbers either clear the bar or they do not. Projects that skip this step launch on enthusiasm and are quietly switched off after the first visible error. Projects that do it launch later, sometimes, and stay in service.
What this means for you
Insist on an evaluation set built from your own cases and labelled by your own people, a bar agreed before building, metrics that include confident wrong answers and adversarial behaviour, and a rerun before every change and every model update. It is the only way to know an AI system works rather than to hope it does, and it is the difference between the systems that stay in service and the demos that did not survive contact with reality.
Frequently asked questions
Why can we not just try it on a few examples?
Because a few examples show that it can work, not how often. AI output varies; the failure rate on the cases nobody tried is what matters in production. An evaluation set of fifty to a few hundred real cases, scored, gives a rate you can rely on and a baseline to compare every later change against.
Who decides what the right answer is?
People who do the work today. They label the evaluation cases once: for each real input, what a good output looks like and what would be unacceptable. That labelling is the most valuable hour the domain experts spend on the project, and it turns their judgement into something the system is measured against forever.
How often do we retest?
Before every change to the prompt, the retrieval or the rules; after every model version change, which happens without your involvement; and on a schedule, monthly, against fresh cases from production. The evaluation set is rerun in minutes; the discipline is remembering that a model update is a change.