This is the second in a series of articles sharing evidence of how technology harms people and the Earth. Most of the evidence cited is less than 24 months old.

Statistic: Facial recognition systems misidentify darker-skinned individuals up to 34% more often than lighter-skinned individuals. (The unseen Black faces of AI algorithms)

“Unsurprisingly, the use of live facial recognition has repeatedly shown itself to be error-prone, with a disconcerting propensity for misidentifying Black people, which has led to a legal challenge. Plans to deploy units more often will simply raise the risk of disastrous and traumatic engagements with people of colour.” – Habib Kadiri, executive director at StopWatch.

The predictive policing feedback loop

Amnesty International’s 2025 report Automated Racism revealed that 32 UK police forces use geographic crime prediction tools trained on historically biased stop-and-search data. These systems disproportionately target neighbourhoods with higher ethnic minority populations, creating a self-fulfilling cycle:

• Over-policing generates more crime reports in minority areas.
• Algorithms interpret this as “high risk,” directing more patrols.
• Increased surveillance leads to more arrests, reinforcing the bias.

In practice, the apparent neutrality of the data is questionable. It has been reported that Durham Police will no longer use postcodes as one of the data points in their model, since it has been argued that doing so perpetuates stereotypes about neighbourhoods that have negative consequences for all residents. (AI profiling: the social and moral hazards of ‘predictive’ policing)

“These systems have been built with discriminatory data and only serve to supercharge racism.” – Sacha Deshmukh, Amnesty International UK

“These technologies have consequences. The future they are creating is one where technology decides that our neighbours are criminals, purely based on the colour of their skin or their socio-economic background.” - Sacha Deshmukh

“These tools to ‘predict crime’ harm us all by treating entire communities as potential criminals, making society more racist and unfair.”  – Sacha Deshmukh

Key issues

• Baked-in inequity: Many clinical algorithms treat racial disparities in health outcomes as biological facts, ignoring how racism and social inequality shape these patterns. For example, use of race in estimating kidney function (eGFR) in nephrology has led to Black patients being less likely to be referred for specialist care. This embeds racial inequities into medical decision-making tools.
• Structural bias: AI models for conditions like skin cancer often perform worse for Black patients, as training datasets overwhelmingly use images of white skin. This increases risks of missed diagnoses and delayed care. (British Medical Journal and other studies)

Quotes

“We need to talk about how society has structural inequality, about people who cannot access healthcare. There are undertones and continuations of structural oppression, discrimination and racism that have reverberations throughout society, and those power dynamics lead to differences in how healthcare is delivered.”
– Dr Joseph Alderman, AI and Digital Health Clinical Research Fellow, University of Birmingham

“There are examples of racial bias in treatment being encoded and fed into algorithms that determine who needs extra care, thereby placing Black people at an even greater disadvantage… We need to educate ourselves on the ways in which data science perpetuates racism.”
– The Lancet: Challenging racism in the use of health data

“Our white paper shows how AI can exacerbate existing health inequities in minority ethnic groups.”
– Dr Saira Ghafur, Institute of Global Health Innovation

“Tackling health inequality is one of the major challenges of our time. Advances in AI and machine learning give us new tools to tackle this challenge, but our enthusiasm must be tempered by a realistic appraisal of the risks of these technologies inadvertently perpetuating inequalities.”
– Lord James O’Shaughnessy, Visiting Professor at the Institute of Global Health Innovation

Key issues

• Algorithmic oppression: Anti-migrant hashtags on X spread 1.66 times faster than factual posts, despite originating from a small user base.
• Exploitative design: TikTok Lite, aimed at global majority users, lacks critical safety features available in its parent app, leaving vulnerable groups exposed to harm.

Quotes

“Algorithmic oppression is not just a glitch in the system but, rather, is fundamental to the operating system of the web.”
– Safiya Umoja Noble, Algorithms of Oppression: How Search Engines Reinforce Racism

“These toxic algorithms are deliberately designed to prioritise engagement and act as incendiaries that fuel division, disinformation, and hate… This model has enabled eye-watering profits for a lucrative Big Tech industry, but has entailed disastrous long-term consequences for human rights, in particular for those most marginalised in society.”
– Pat de Brún, Amnesty International

“Social media algorithms prioritise engagement, so divisive content spreads faster. In the context of anti-migrant rhetoric, this becomes a tool for radicalisation.”
– Pat de Brún, Amnesty International

Key issues

• Exclusion by design: Algorithms penalise candidates from historically marginalised backgrounds, filtering out those with non-Anglo-Saxon names or affiliations with racial justice movements.
• Lack of accountability: Few studies empirically assess the extent of bias in these systems, leaving harmed communities without recourse.

Quotes

“Various examples of algorithmic bias in recruitment systems are becoming apparent, although there are relatively few studies which have assessed the extent of this bias empirically.”
– Bias in Algorithmic Decision-Making, Centre for Data Ethics and Innovation

“There is now an overwhelming body of research demonstrating how the growing reliance on AI and automated systems has increased inequity, discrimination and marginalisation.”
– Transforming Society, June 2024

“The growing use of machine learning in public and private sector services will further disadvantage digitally excluded groups, who are often poorly represented in datasets and are likely to face further marginalisation as a result.”
– Digital exclusion report, UK Government Communications and Digital Committee

Key issues

• Exploitative AI: Self-teaching algorithms in banking and insurance absorb societal biases, disproportionately harming Black and brown customers.
• Lack of transparency: Institutions rarely explain how AI makes decisions, leaving victims unable to challenge discriminatory outcomes.

Quotes

“It is now a commonplace in artificial intelligence and machine learning that algorithms that govern mortgage lending [and] insurance quotes…are biased against women, and even more so against people of colour. Black people pay billions in extra premiums and higher loan rates… But no one quite knows how and why the machines learn to discriminate, much less how to stop them.”
– Trevor Philips, the former chair of the Equality and Human Rights Council in Landscape Summary: Bias in algorithmic decision making

“Racist and sexist artificial intelligence (AI) bots pose a risk to the financial system… Self-teaching algorithms could pick up biases from datasets and wider society, which could then discriminate against customers or staff in the workplace.”
– Kathleen Blake, lead analyst, Bank of England

The convergence of data from UK policing, healthcare, and employment systems paints an irrefutable picture: algorithmic racism is not incidental but institutional. Predictive policing tools, biased medical AI, and exploitative financial practices all stem from the same root – a tech industry built on colonial hierarchies and profit-driven indifference.

As Charlene Prempeh argues, “Technology’s been coming for Black people since the start. The question is whether we’ll let it define our future.” The path forward demands abolishing systems that conflate risk with race and rebuilding tech governance around principles of equity, transparency, and repair.

“Better never means better for everyone… It always means worse, for some. We must centre those ‘some’ in our solutions.”
– Dr. Ruha Benjamin

Read our evidence article #1: Why digital exclusion is a tech justice issue.

You can also read more about tech injustice in our Tech Justice Report 2023 and how we can end it in Catalyst’s ‘Manifesto for a Tech Just World’. Or see some examples of tech justice in action in ‘What is tech justice?‘ and the great work done by Catalyst network members in the Tech Justice Roadtrip project.

Get notified of our next article sharing evidence of how technology harms people and the Earth by signing up for our newsletter.