Bias in AI Decision-Making: What Senior HR Leaders Need to Know

Bias in AI decision-making is one of the most pressing issues and risks for business implementing AI-based HR solutions.  This article is an overview of how it happens, what it means under UK law, and what organisations need to do now to take control of the risks.

What do we mean by AI bias in HR?

Bias in AI systems is often misunderstood. It’s not about the software being malicious or deliberately discriminatory. In most cases, it’s the opposite — the system is functioning exactly as it was trained to. But AI models learn from data. If the training data reflects biased decisions, incomplete records, or historical inequalities then the system will replicate and even amplify those biases.

Consider a recruitment tool trained on ten years of CVs from successful applicants. If over that period, the majority of hires were men, or people from similar socio-economic backgrounds, the model may learn to prioritise those profiles — not because they’re better candidates, but because that’s what success looked like historically. That’s not innovation. That’s automating discrimination.  Even worse, it hardwires discrimination into the system.  Most AI systems work as a feedback loop, learning from the decisions it is taking, meaning that its decisions may get more discriminatory over time, not less.

These aren’t hypothetical problems. Amazon famously scrapped an AI recruiting tool after discovering it consistently downgraded CVs that included the word “women’s”, such as “women’s chess club captain”.  In the US, a healthcare algorithm used to identify high-risk patients was found to be racially biased, underestimating the needs of black patients due to flawed cost-based assumptions. In the UK, evidence is mounting that similar risks are emerging in HR technology — quietly and often invisibly.

The legal landscape in the UK

UK law does not have any AI-specific legislation that directly regulates bias in HR technology. But that doesn’t mean it’s a free-for-all. The Equality Act 2010 remains central. If an AI tool used in recruitment, appraisal, or redundancy selection leads to a discriminatory decision then your organisation could be liable.

Not all bias will be unlawful discrimination. 

That bias must be based on a protected characteristic.  That may be sex, sexual orientation, race, religion or belief, disability, transgender status, marriage or maternity status.  It is only if a decision is taken based upon one of these protected characteristics that the equality legislation is triggered.

Second, we need to remember the two main types of discrimination: direct discrimination and indirect discrimination.

Direct discrimination occurs when a person is treated less favourably than another person because of a protected characteristic.

Indirect discrimination occurs when the employer applies a provision, criterion or practice (PCP) that disproportionately disadvantages person because they are a member of a protected group.

Indirect discrimination is unlawful only if the employer cannot show that the PCP is justified as a proportionate means of achieving a legitimate aim. In contrast direct discrimination can only very rarely be lawful.

It is likely that we will see challenges to decisions taken by or with the support of AI systems with assertions of both direct and indirect discrimination.

All employment decisions are open to challenge as being unlawfully discriminatory – the candidate who did not get the job; the employee passed over for promotion; the dispute over a bonus; redundancy selection.  AI beings a further dimension to those challenges.

This area also triggers data protection legislation.  A key principle of the UK GDPR is that data must be processed “fairly”.  It is difficult to see that discriminatory processing will be fair.  We are also likely to see the data access provisions of the data protection legislation triggered at an early stage of disputes, thus bringing data protection law very firmly into the employment law world.

Where bias creeps in

Bias can enter an AI system at any point. It might be hidden in the training data, where past decisions have been skewed towards or against a particular factor.  It might be in the design of the model itself, if certain variables are given more weight than they deserve. It might come from the way the system is deployed: if a tool is used differently across departments or managers, results can vary wildly, sometimes with discriminatory outcomes.

Transparency is often lacking. Many AI systems are “black boxes”, offering results without explanations. This makes bias hard to detect and even harder to challenge. A rejected candidate may never know that the algorithm penalised them for having a gap in employment, even if that gap is due to maternity leave or a disability-related career break. If no one in your organisation understands how the system makes decisions, you can’t defend it, explain it or justify it, and it will not be a sufficient defence to blame the AI system.

Accountability without understanding is risky

One of the biggest dangers right now is the false sense of confidence that can come from AI. There’s a temptation to see it as objective, neutral, even smarter than humans. But as any data scientist will tell you, AI is only as fair as the data and logic that feed it.

There’s a growing expectation that employers must be able to explain, audit, and justify AI-driven decisions, especially where they can have a significant impact on people’s careers.

Procurement teams need to start asking harder questions of vendors. What data was the model trained on? Has it been tested for bias across different demographic groups? Can decisions be explained to candidates or employees? Is there a human in the loop who can review or override the system?

So, what should HR leaders do now?

Start by owning the issue. AI is not an IT problem or a procurement matter. It’s a people issue — and that makes it a core HR responsibility. You don’t need to be a data scientist, but you do need to understand the principles and ask the right questions.

Push for explainability. If a system can’t explain how it reached a decision, don’t trust it. Favour suppliers who offer transparency and who will work with you to audit outcomes regularly.

Bias detection isn’t a checkbox at the end — it needs to be built into how systems are selected, configured, and monitored.

Train your teams. Hiring managers and HR staff need to understand both the capabilities and the limits of AI tools. Blind faith is as dangerous as outright resistance.

And finally, keep the human in the loop. AI can support decisions, but it should never replace accountability. People’s lives and careers are too important to delegate to a machine.

Looking Ahead

Bias in AI creates a new paradigm for discrimination claims and anti-discrimination strategies in the workplace.  Not only will employment tribunals have to get used to the concept, but employment tribunals will not be the only forum for challenging potentially discriminatory decisions.  The Information Commissioner’s Office has a key role to play, and HR teams will have to polish up their knowledge of data protection law.

This is the beginning of a longer conversation. In the next in this series, we’ll dig deeper into how to audit AI systems for bias in practice, and what meaningful transparency really looks like in an HR context.

For now, the message is simple: bias in AI HR systems isn’t just a tech problem. It’s a people risk – and people risks are HR’s domain. Senior leaders must lead the way in ensuring that AI works for people, not against them.

Contact us

You can contact Matthew Cole for assistance on any of these issues by e-mailing Matthew at mcole@prettys.co.uk