Bias in AI systems: how it happens and how to check

The short answer

Bias in an AI system is not usually a matter of intent. It enters through three doors. The data, which records how decisions were made before, including inequalities nobody designed, and which often has fewer or poorer records for some groups. The design, which chooses what to optimise for, and optimising for past success reproduces past patterns. And use, where people defer to a system’s output because it looks objective, which converts a suggestion into a decision nobody examined. Removing sensitive attributes from the data does not solve it, because postcode, name, school, employment gaps, language and photographs all stand in for them; it only removes your ability to detect the problem. The practical approach for a small business is outcome testing: run representative cases that differ only in the attribute you are checking, compare the results, then look at real outcomes over a quarter grouped by those attributes, and investigate any pattern without a business explanation. Where AI touches decisions about people, keep a human deciding, document the criteria and be able to explain individual outcomes.

Where bias comes from and what to do

Source	How it happens	What to do
Historical data	The system learns from past decisions, including biased ones	Test outcomes; consider whether past decisions are a valid target at all
Unrepresentative data	Fewer or lower-quality records for some groups, so performance is worse for them	Measure accuracy per group, not overall
Proxy variables	Postcode, name, school, career gaps stand in for protected attributes	Do not rely on field removal; test outcomes
Objective choice	Optimising for a metric that correlates with a group	Ask what the metric rewards and who it disadvantages
Labelling	The examples were labelled by people with their own patterns	Review labels; use several labellers on samples
Deference in use	Staff treat the output as fact because it looks objective	Show uncertainty; require a reason for the decision; train people
Feedback loops	The system’s outputs shape future data, entrenching the pattern	Monitor drift; retest periodically

A practical check for a small business

What to do when you find something

Do not delete the evidence. Investigate what drove the difference, decide whether it reflects a legitimate business factor, and if not, change the criteria, the data or the process. Record what you found and what you changed; a documented finding and fix is a defensible position, while an undocumented suspicion is not. If the system cannot be made to treat people alike, stop using it for that decision. That is a legitimate outcome of testing, and it is cheaper than the alternative.

What this means for you

Bias arrives through data, design and deference, not intent, and removing sensitive fields hides it rather than fixing it. Test outcomes with representative cases and quarterly reviews across groups, keep the attributes available for testing, investigate patterns, keep a person deciding with documented criteria, and record what you checked. For a small business that is a morning’s work per system, and it is the difference between a defensible process and an indefensible one. This is general information rather than legal advice.