Explainability: when you must be able to explain a decision
When a business must explain an AI-influenced decision, what an adequate explanation contains, and how to design for it.
The short answer
If a decision affects a person’s rights, money or opportunities, you should be able to explain it to them in terms they understand, regardless of what technology was involved in reaching it. That is the practical standard, and it is broadly what European data protection law and the AI Act require in their respective areas. An adequate explanation is not a description of a model. It says what was decided, what information it was based on, which criteria were applied, who made the decision, and how to ask for a review. That means explainability is a design choice made before the system is built: keep the decision criteria explicit rather than implicit in a model, record the inputs and the stated reason at the moment of the decision, and keep a human decision-maker who can give the reason independently. A system that cannot produce this should not be used to decide about people, however accurate it appears, because the day someone asks why, and they will, the answer has to exist.
What an explanation must contain
| Element | Example | Why |
|---|---|---|
| What was decided | Your application was declined | The person needs the outcome stated plainly |
| On what information | Based on the address, the requested date and your stated requirements | Shows what was considered, and lets them correct errors |
| Against which criteria | Our service area and availability rules, which are published here | Criteria must exist and be stateable |
| Who decided | Reviewed and decided by our operations manager | Accountability, and required where automated decisions are restricted |
| How to challenge | Reply to this message or call, and we will review with fresh eyes | The right to contest is the point of the explanation |
| What would change it | If the property is within the area after all, or another date suits | Makes the explanation useful rather than final |
Designing for explainability
- Write the criteria for every decision AI touches, before building anything.
- Use the model to support, not to decide: it may surface information or suggest, but the criteria decide.
- Record at the moment of decision: inputs used, suggestion given, criteria applied, reason stated, person responsible, timestamp.
- Draft the explanation template for each decision type, in plain language.
- Test it on real cases with someone outside the process: would they understand and know how to challenge?
- Provide the challenge route and staff it.
- Review the explanations given quarterly; unclear ones point at unclear criteria.
Where this bites in practice
Hiring and shortlisting. Credit, payment terms and deposits. Pricing that differs between customers. Eligibility for a service, a discount or support. Tenant and applicant selection. Insurance and claims. Content moderation and account suspension. Prioritisation in queues where waiting has consequences. In each, the person on the receiving end has a reasonable expectation of knowing why, and increasingly a legal right to it.
What this means for you
Be able to explain any decision that affects a person: what was decided, on what information, against which criteria, by whom, and how to challenge it. Design for that by writing criteria in advance, keeping the model in a supporting role, recording the reason at the moment of decision and keeping a human decision-maker. If a system cannot support that, do not use it for decisions about people.
Frequently asked questions
Does the law require us to explain AI decisions?
European data protection law gives people rights around solely automated decisions with significant effects, including meaningful information about the logic involved and the right to human intervention. The AI Act adds obligations for high-risk systems. Beyond the law, clients, insurers and the affected person all expect an explanation. The practical rule is that if a decision matters to someone, be able to explain it.
What does a good explanation look like?
Plain language covering five things: what was decided, what information it was based on, which criteria were applied, who made the decision, and how to ask for a review. For example: your application was declined because the property falls outside our service area and the requested date is unavailable; this was reviewed by our operations manager; here is how to ask us to look again. Notice that no model is mentioned; the person needs the reasons, not the architecture.
What if the model cannot be explained?
Then it is the wrong tool for that decision. Complex models can be used to support a decision where a person applies explicit criteria and can state the reasons independently of the model, but they cannot be the reason itself. If the only honest answer is that the system produced a score, do not use it to decide about people. That is a design constraint, not a limitation to work around.
Sources
- European Commission: AI Act (accessed 2026-09-12)