Quarterly AI reviews: what to measure and adjust
One hour a quarter keeps AI owned, measured and honest. The fixed agenda, the scorecard, the decisions, and when to switch a system off.
The short answer
AI in a small business stays owned, measured and honest with one hour a quarter, run to a fixed agenda by the accountable person, with each system’s owner reporting in a line or two. The hour reads the inventory, the scorecard, the quality samples, the incidents, the costs, the risk register and the vendor changes, and it makes decisions: keep, improve, expand, pause or retire each system, and which candidate becomes the next project. The written record of those hours is the governance history: what you ran, what it did, what you decided and why. It is the document that answers auditors, clients and your own questions a year from now.
The agenda
| Item | Input | Decision |
|---|---|---|
| Inventory | New systems or AI features enabled since last quarter; anything retired | Add to the register with an owner, or remove |
| Scorecard | One line per system: business outcome, process metrics, quality sample, adoption | Keep, improve, expand, pause or retire |
| Quality | Monthly sample scores, confident-wrong rate, escalation trend, model version changes | Rerun evaluation, adjust prompts or documents, raise or lower the gate |
| Incidents | What went wrong, how it was caught, what changed | Update controls in the register |
| Costs | Total, cost per case, versus forecast; cap breaches | Adjust caps, model choices, forecast |
| Risks | Register review: new risks, controls still in place, gaps with dates | Assign, re-date, close |
| Vendors | Announcements: retirements, price and terms changes; alternative last tested | Schedule a switch test or a switch |
| Next project | Candidates against the criteria; data readiness | Choose one, name an owner, set the four-week plan |
The decisions, and what triggers them
- Keep: outcome moving or holding, quality stable, cost per case flat. Two minutes.
- Improve: quality slipping or escalations rising with a known cause. Owner gets a task and a date.
- Expand: outcome strong, gate widening safely, adjacent process obvious. Becomes a candidate.
- Pause: an incident, a terms change or a quality drop with no immediate fix. Switched off until resolved.
- Retire: no movement in the business outcome after six months, quality falling with no owner able to fix it, cost per case rising without reason, or the process it served has changed. Documented and switched off.
The record
One page per quarter: the agenda items, the numbers, the decisions, the owners and dates. Stored with the governance documents. Over a year it shows a business that knows what it runs, measures it, acts on the measurements and can explain every system’s history. That record is worth more to a client questionnaire or an audit than any policy, because it shows the policy being lived.
What this means for you
One hour, four times a year, fixed agenda, prepared scorecard, real decisions, written record. It keeps every AI system owned and measured, retires the ones that stopped earning their place, catches drift and vendor changes before they cost anything, and chooses the next project from evidence. It is the smallest governance habit with the largest effect, and it fits in the calendar of any business that runs AI at all.
Frequently asked questions
One hour is enough?
For a small business with a handful of systems, yes, if the scorecard and the owners' one-line reports are prepared in advance. The hour is for reading, deciding and recording, not for discovering. If it regularly runs long, either the preparation is missing or there are more systems than owners, and both are findings.
What if a system is fine and nothing needs deciding?
Then the decision is keep, recorded with the numbers that justify it, and the line takes two minutes. Reviews are not obliged to change things; they are obliged to look. A system that is fine for four quarters in a row is a success story worth noting, and its owner has earned the shorter report.
Who attends?
The accountable person, each system's internal owner, and your partner if one runs the systems, for the technical items. Not a committee: the people who can answer the questions and the one who decides. Finance for the cost line if it is material.