Fair Access

Introduction

Medical AI is often presented as a route to healthier, longer lives at population scale: faster diagnoses, personalised treatments, lower costs, and eventually a health system that can respond earlier rather than waiting for severe illness. But the central fairness question is simple and uncomfortable: who benefits first, and who gets missed?

Fair Access illustration 1 The risk is not only that AI systems sometimes fail. It is that they fail unevenly. A diagnostic model trained mostly on patients from wealthy hospitals may work less well for poorer communities, rural clinics, women, minority ethnic groups, disabled patients, or people with uncommon conditions. A healthcare system that relies heavily on smartphones, wearable devices, or high-quality electronic records may systematically exclude people who lack those tools. And if advanced medical AI becomes expensive infrastructure controlled by a small number of firms or health systems, the gains could widen existing health gaps rather than close them.

This matters far beyond current hospital software. The optimistic “AI bloom” case depends on broad human flourishing: longer healthy lives across societies, not merely technological progress for already advantaged populations. Medical AI becomes a genuine civilisational gain only if validation, monitoring, affordability, and access work across real populations and real health systems.

The people missing from the training data

Many medical AI systems learn from historical healthcare data: scans, electronic health records, lab tests, insurance claims, or genomic databases. The problem is that healthcare data is not a neutral snapshot of humanity. It reflects who had access to care in the first place, who was diagnosed correctly, which hospitals digitised records early, and which populations were heavily studied.

That means some groups appear far more often in training datasets than others.

Researchers and regulators have repeatedly warned that under-representation can produce systems that perform worse for excluded populations. The STANDING Together recommendations, published in The Lancet Digital Health, argue that medical AI datasets need much greater transparency about who is represented and who is missing because minority populations are often under-represented in training and evaluation data. [The Lancet]thelancet.comThe LancetTackling algorithmic bias and promoting transparency in…by JE Alderman · 2025 · Cited by 131 — Tackling algorithmic bias and…

This problem appears across several layers of medicine:

Skin tone and imaging: Dermatology AI systems trained mostly on lighter skin can miss disease on darker skin. The same concern applies to wound assessment and some imaging workflows.
Women’s health: Historically male-biased datasets can reduce sensitivity to symptoms or disease presentations more common in women.
Rare diseases: AI systems often work best on common conditions with abundant data, not unusual or poorly documented disorders.
Low-income countries: Many models are trained primarily on data from North America or Europe, then deployed elsewhere with very different disease patterns and clinical infrastructure.
Genomics: Large genetic databases remain heavily skewed toward people of European ancestry, limiting the reliability of AI-assisted risk prediction for many populations. [The Guardian]theguardian.comThe Guardian UK report reveals bias within medical tools and devicesIt emphasizes the need for an equity perspective throughout the lifecycle of medical devices to ensure fair healthcare. Concerns were hig…

This is not merely a technical nuisance. If medical AI becomes deeply integrated into screening, triage, and resource allocation, unequal performance could shape who receives early treatment, specialist referrals, or preventive care.

When biased devices become AI inputs

One of the clearest examples comes from pulse oximeters, the small devices clipped onto a finger to estimate blood oxygen levels.

During the COVID-19 pandemic, researchers found that pulse oximeters were more likely to miss dangerously low oxygen levels in Black patients. A widely cited New England Journal of Medicine study found occult hypoxaemia — hidden low oxygen saturation — occurred far more often in Black patients than White patients when pulse oximeters showed apparently reassuring readings. [New England Journal of Medicine]nejm.orgNew England Journal of MedicineRacial Bias in Pulse Oximetry Measurementby MW Sjoding · 2020 · Cited by 1242 — The question of whether pu…

The problem likely emerged because devices were calibrated disproportionately on lighter skin tones. Subsequent reporting and regulatory reviews highlighted the broader implications for healthcare equity. [The Guardian]theguardian.comThe Guardian UK report reveals bias within medical tools and devicesIt emphasizes the need for an equity perspective throughout the lifecycle of medical devices to ensure fair healthcare. Concerns were hig… [Oncology Nursing Society]ons.orgfda says its continuing evaluate pulse oximetersOncology Nursing SocietyFDA Says It's Continuing to Evaluate Pulse Oximeters…Jun 27, 2022 — More than a year and a half after a report…

This matters for AI because modern medical systems increasingly chain technologies together. A biased sensor can feed biased data into a machine-learning model, which then influences triage or treatment decisions.

Researchers studying downstream effects found that pulse oximetry bias can reduce the ability of machine-learning systems to correctly predict severe outcomes in affected patients, effectively reproducing false reassurance inside AI workflows. [arXiv]arxiv.orgSource details in endnotes.

The lesson is broader than pulse oximeters. Medical AI is not only shaped by algorithms. It is shaped by the quality and representativeness of the entire measurement system underneath it.

Scaling can amplify existing inequalities

Medical AI often works best in well-funded hospitals with clean digital infrastructure, abundant specialist oversight, and high-quality data pipelines. But those are already the environments with the best health outcomes.

This creates a paradox. AI could theoretically democratise expertise by helping overstretched clinics and lower-income countries. Yet the first large gains may accrue mainly to elite systems able to afford integration, monitoring, cybersecurity, legal compliance, and continual model updates.

Several forms of exclusion matter simultaneously:

Digital exclusion

Many AI-enabled health systems assume reliable internet access, smartphones, wearable devices, or patient portals. Older adults, poorer households, migrants, and people with limited digital literacy may struggle to participate.

An AI healthcare system built around continuous monitoring and digital engagement can unintentionally reward people already easiest to reach.

Infrastructure inequality

Some hospitals can afford advanced imaging systems, cloud computing, and specialist AI oversight teams. Others cannot.

A rural clinic with intermittent connectivity and staff shortages cannot deploy AI the same way as a major research hospital in London, Boston, or Singapore. The danger is a two-tier system where AI increasingly concentrates expertise and investment in already strong institutions.

Insurance and reimbursement gaps

In mixed or privatised systems, insurers may reimburse profitable AI-supported services faster than less lucrative areas of care. Cosmetic or efficiency-focused applications can spread more rapidly than labour-intensive chronic disease management.

Even in public systems such as the NHS, budget pressure creates difficult questions about which AI tools justify nationwide deployment.

Language and cultural mismatch

Large language models used in healthcare may perform worse across accents, dialects, languages, or culturally specific forms of communication. Symptoms are described differently across populations. Medical trust also differs between communities.

An apparently “universal” AI assistant can become much less reliable once it leaves the population it was optimised for.

Foundation models may reproduce historical prejudice

Newer medical AI systems increasingly rely on large “foundation models”: broad AI systems trained on enormous datasets and adapted to many tasks. These models may eventually support diagnosis, documentation, research, and patient communication at scale.

But foundation models can inherit patterns from historical healthcare systems.

A 2024 Lancet Digital Health study examining GPT-4 in healthcare contexts found evidence of racial and gender bias in generated medical responses. [The Lancet]thelancet.comThe LancetTackling algorithmic bias and promoting transparency in…by JE Alderman · 2025 · Cited by 131 — Tackling algorithmic bias and…

Researchers studying medical imaging foundation models similarly found that some systems underdiagnosed historically marginalised groups, including Black patients and women, across multiple datasets. [arXiv]arxiv.orgSource details in endnotes.

The deeper concern is not simply offensive outputs. It is statistical invisibility.

If historical healthcare systems diagnosed some groups later, studied them less, or recorded their symptoms differently, AI systems trained on those records can absorb and normalise those patterns. In effect, automation may freeze existing inequalities into future infrastructure.

That is one reason many researchers now argue that fairness cannot be treated as a final “bias correction” layer added after deployment. It has to shape data collection, evaluation standards, and monitoring from the start. [ScienceDirect]sciencedirect.comScienceDirectAddressing algorithmic bias and the perpetuation of health…by R Agarwal · 2023 · Cited by 208 — In the absence of awarene…

Fair Access illustration 2

Real-world monitoring matters more than benchmark scores

Medical AI often looks impressive in controlled studies. The harder question is what happens after nationwide deployment.

A system may perform well during testing but degrade once clinical practices, patient demographics, or disease patterns change. Regulators increasingly describe this as “data drift” or “model drift”. [U.S. Food and Drug Administration]fda.govU.S. Food and Drug AdministrationWe Regulate Food Drugs, Medical Devices, Radiation-Emitting Products, Vaccines, Blood, and Biologics, An…

This matters because medical AI systems are not static medicines. They interact continuously with changing hospitals, changing patients, and changing workflows.

Real-world deployment can expose hidden weaknesses:

A triage model trained in one region may fail elsewhere.
A diagnostic assistant may become less accurate as disease prevalence changes.
Clinicians may over-trust automated suggestions.
Busy staff may use tools differently from the conditions assumed in trials.

Recent reporting on AI-assisted surgical systems illustrates the stakes. Reuters documented adverse-event reports involving AI-enabled medical devices, including claims of software misidentification and navigation failures. The reporting also highlighted concerns that regulators may struggle to keep pace with rapidly expanding AI deployment. [Reuters]reuters.comIntroduced by Acclarent, a Johnson & Johnson unit, the AI upgrade to TruDi coincided with a sharp increase in reported malfunctions and i…

Not every reported incident reflects confirmed AI fault. But the broader lesson is important: healthcare AI needs continuous post-deployment monitoring, especially across diverse populations.

A medical AI system should not be considered “fair” simply because it passed an initial approval process.

Affordability may decide whether AI medicine becomes a public good

The optimistic case for AI medicine assumes abundance: cheaper diagnostics, faster drug discovery, wider access to expertise, and eventually lower healthcare costs.

That outcome is possible, but not automatic.

Advanced AI systems require expensive computing infrastructure, proprietary datasets, cybersecurity, specialist staff, and regulatory compliance. If these costs remain concentrated among a few firms or wealthy health systems, AI medicine could deepen dependence rather than create broad abundance.

Several distribution questions matter:

Will lifesaving AI tools be affordable in lower-income countries?

Will public health systems own core infrastructure, or rent access from a handful of companies?
Will open scientific models exist alongside proprietary systems?
Will rural and community clinics receive deployment support?
Will AI reduce clinician shortages, or mainly increase productivity expectations for already stretched staff?

The political economy matters because health inequality compounds over time. If affluent populations gain earlier access to AI-assisted prevention, genomic medicine, and longevity technologies, healthy-life expectancy gaps could widen substantially.

In the most optimistic AI bloom scenario, medical intelligence becomes widely available infrastructure — closer to public sanitation or vaccines than luxury medicine. But reaching that outcome requires policy choices, procurement choices, and international coordination, not merely technological progress.

Fair Access illustration 3

What a fairer version of medical AI would require

A more equitable model of medical AI is technically possible, but it requires deliberate design rather than optimistic assumptions.

Several principles increasingly appear across research and regulatory discussions:

Better representation in datasets

Developers need datasets that include varied ages, ancestries, disabilities, socioeconomic groups, languages, and clinical settings. That also means documenting who is absent.

Performance reporting by subgroup

A single overall accuracy score can hide unequal outcomes. Systems increasingly need subgroup evaluation across race, sex, age, geography, and other relevant factors.

Continuous monitoring after deployment

Healthcare AI should be treated as evolving infrastructure. Real-world surveillance is essential, especially when models update over time.

Public-sector capability

Health systems need enough in-house technical expertise to evaluate vendor claims, audit systems, and negotiate from a position of strength.

Global inclusion

If AI medicine is trained almost entirely on wealthy-country data, the long-term benefits will remain geographically narrow. Building representative global datasets is difficult but important.

Human oversight and contestability

Patients and clinicians need ways to question, override, and investigate AI outputs. Automation without accountability can make errors harder to detect rather than easier.

The long-term bloom case depends on distribution, not just capability

The strongest version of the AI bloom argument is not simply that medicine becomes more technologically advanced. It is that humanity gains the ability to reduce suffering and extend healthy life at unprecedented scale.

But the history of medicine shows that breakthroughs do not spread automatically. Vaccines, antibiotics, maternal care, and clean water transformed human life because institutions, public investment, and political decisions widened access over time.

Medical AI faces the same test.

A future with AI-assisted longevity, continuous health monitoring, personalised treatment, and rapid scientific discovery could dramatically expand human flourishing. Yet it could also produce a world where elite populations receive increasingly preventive, data-rich care while poorer populations remain stuck with overstretched systems and delayed treatment.

The question “who gets left out?” is therefore not a side issue to the optimistic vision. It is one of the central conditions determining whether AI medicine becomes a broad civilisational gain or another layer of uneven technological advantage.

Endnotes

Source: arxiv.org
Link: https://arxiv.org/abs/2604.14514
Source: arxiv.org
Link: https://arxiv.org/abs/2408.04396
Source snippet
arXivEvaluating the Impact of Pulse Oximetry Bias in Machine Learning under Counterfactual ThinkingAugust 8, 2024...

Published: August 8, 2024
Source: arxiv.org
Link: https://arxiv.org/abs/2402.14815
Source snippet
arXivDemographic Bias of Expert-Level Vision-Language Foundation Models in Medical ImagingFebruary 22, 2024...

Published: February 22, 2024
Source: sciencedirect.com
Link: https://www.sciencedirect.com/science/article/abs/pii/S2211883722001095
Source snippet
ScienceDirectAddressing algorithmic bias and the perpetuation of health...by R Agarwal · 2023 · Cited by 208 — In the absence of awarene...
Source: sciencedirect.com
Link: https://www.sciencedirect.com/science/article/pii/S2589750025001396
Source snippet
A practical framework for operationalising responsible and...by ML Welch · 2026 — This framework aligns with reporting standards such as...
Source: fda.gov
Link: https://www.fda.gov/medical-devices/digital-health-center-excellence/request-public-comment-measuring-and-evaluating-artificial-intelligence-enabled-medical-device
Source snippet
Food and Drug AdministrationEvaluating AI-enabled Medical Device Performance in...30 Sept 2025 — AI system performance can be influenced...
Source: fda.gov
Link: https://www.fda.gov/medical-devices/medical-device-regulatory-science-research-programs-conducted-osel/performance-evaluation-methods-evolving-artificial-intelligence-ai-enabled-medical-devices
Source snippet
Food and Drug AdministrationPerformance Evaluation Methods for Evolving Artificial...10 Jun 2024 — The goal of this regulatory science r...
Source: reuters.com
Link: https://www.reuters.com/investigations/ai-enters-operating-room-reports-arise-botched-surgeries-misidentified-body-2026-02-09/
Source snippet
Introduced by Acclarent, a Johnson & Johnson unit, the AI upgrade to TruDi coincided with a sharp increase in reported malfunctions and i...
Source: sciencedirect.com
Link: https://www.sciencedirect.com/science/article/pii/S2772963X2500599X
Source snippet
Analysis of FDA-Approved Artificial Intelligence and...by P Sardar · 2025 · Cited by 4 — Despite the growing number of market-approved m...
Source: sciencedirect.com
Link: https://www.sciencedirect.com/science/article/abs/pii/S0925231226003164
Source snippet
AI models show dramatic AUC drops (0.95 → 0.63) on...Read more...
Source: sciencedirect.com
Link: https://www.sciencedirect.com/science/article/pii/S2589750024002243
Source snippet
Dissecting racial bias in an algorithm used to manage the health of populations.Read mor...
Source: research.uhb.nhs.uk
Link: https://www.research.uhb.nhs.uk/new-recommendations-to-increase-transparency-and-tackle-potential-bias-in-medical-ai-technologies/
Source snippet
recommendations to increase transparency and...18 Dec 2024 — People who are in minority groups are particularly likely to be under-repre...
Source: thelancet.com
Link: https://www.thelancet.com/journals/landig/article/PIIS2589-7500%2824%2900224-3/fulltext
Source snippet
The LancetTackling algorithmic bias and promoting transparency in...by JE Alderman · 2025 · Cited by 131 — Tackling algorithmic bias and...
Source: theguardian.com
Title: The Guardian UK report reveals bias within medical tools and devices
Link: https://www.theguardian.com/society/2024/mar/11/medical-tools-devices-healthcare-bias-uk
Source snippet
It emphasizes the need for an equity perspective throughout the lifecycle of medical devices to ensure fair healthcare. Concerns were hig...
Source: nejm.org
Link: https://www.nejm.org/doi/full/10.1056/NEJMc2029240
Source snippet
New England Journal of MedicineRacial Bias in Pulse Oximetry Measurementby MW Sjoding · 2020 · Cited by 1242 — The question of whether pu...
Source: ons.org
Title: fda says its continuing evaluate pulse oximeters
Link: https://www.ons.org/publications-research/voice/news-views/06-2022/fda-says-its-continuing-evaluate-pulse-oximeters
Source snippet
Oncology Nursing SocietyFDA Says It's Continuing to Evaluate Pulse Oximeters...Jun 27, 2022 — More than a year and a half after a report...
Source: thelancet.com
Link: https://www.thelancet.com/journals/landig/article/PIIS2589-7500%2823%2900225-X/fulltext
Source snippet
Methods. Using the Azure OpenAI application interface...Read more...
Source: fda.gov
Link: https://www.fda.gov/
Source snippet
U.S. Food and Drug AdministrationWe Regulate Food Drugs, Medical Devices, Radiation-Emitting Products, Vaccines, Blood, and Biologics, An...
Source: thelancet.com
Link: https://www.thelancet.com/journals/landig/article/PIIS2589-7500%2825%2900139-6/fulltext?rss=yes
Source snippet
tackling bias, inequity, and implementation challenges20 Mar 2026 — We critically discuss the challenges associated with the use of such...
Source: thelancet.com
Link: https://www.thelancet.com/journals/landig/article/PIIS2589-7500%2825%2900003-2/fulltext
Source snippet
Weighing the benefits and risks of collecting race...by A Fiske · 2025 · Cited by 8 — In this study, AI developers assumed that health-c...
Source: linkedin.com
Link: https://www.linkedin.com/posts/the-lancet_experts-have-developed-recommendations-activity-7282804003283861506-yT1d

Additional References

Source: origin-cdrh-rst.fda.gov
Link: https://origin-cdrh-rst.fda.gov/biasmytireport
Source snippet
| Center for Devices and Radiological HealthThis regulatory science tool presents two methods to amplify Artificial Intelligence (AI) / M...
Source: researchgate.net
Link: https://www.researchgate.net/publication/347658013_Racial_Bias_in_Pulse_Oximetry_Measurement
Source snippet
(PDF) Racial Bias in Pulse Oximetry MeasurementAs demonstrated by Sjoding et al., because the algorithms for these devices were primarily...
Source: futuremedicine.com
Link: https://www.futuremedicine.com/articles/ai-bias-a-problem-long-before-the-algorithm
Source snippet
AI Bias: A Problem Long Before the AlgorithmIf not addressed, AI biases could "poison" public and institutional perceptions of digital he...
Source: birmingham.ac.uk
Link: https://www.birmingham.ac.uk/news/2024/new-recommendations-to-increase-transparency-and-tackle-potential-bias-in-medical-ai-technologies
Source snippet
University of BirminghamNew Recommendations to Increase Transparency and...18 Dec 2024 — People who are in minority groups are particula...
Source: nhsrho.org
Link: https://nhsrho.org/research/pulse-oximetry-and-racial-bias-recommendations-for-national-healthcare-regulatory-and-research-bodies/
Source snippet
Pulse Oximetry and Racial Bias: Recommendations for...17 Apr 2023 — It recommended that the NHS Race and Health Observatory (NHSRHO) und...
Source: GOV.UK
Link: https://www.gov.uk/government/publications/the-regulation-of-artificial-intelligence-as-a-medical-device-government-response-to-the-rhc/the-regulation-of-artificial-intelligence-as-a-medical-device-government-response-to-the-regulatory-horizons-council
Source snippet
regulation of artificial intelligence as a medical device10 Mar 2025 — The UK medical devices regulations require that products are desig...
Source: fenwick.com
Link: https://www.fenwick.com/insights/publications/fda-issues-draft-guidances-on-ai-in-medical-devices-drug-development-what-manufacturers-and-sponsors-need-to-know
Source snippet
The FDA's bias control concerns focus on minimizing demographic biases in training data to ensure that a device benefits all...Read more...
Source: fda.gov
Title: artificial intelligence program research aiml based medical devices
Link: https://www.fda.gov/medical-devices/medical-device-regulatory-science-research-programs-conducted-osel/artificial-intelligence-program-research-aiml-based-medical-devices
Source snippet
Artificial Intelligence Program: Research on AI/ML-Based...26 Sept 2024 — AI technologies are transforming health care by producing diag...
Source: hai.stanford.edu
Title: HAI Policy Brief Toward Stronger FDA Approval Standards for AI Medical Devices 1
Link: https://hai.stanford.edu/assets/files/2022-06/HAI%20Policy%20Brief%20-%20Toward%20Stronger%20FDA%20Approval%20Standards%20for%20AI%20Medical%20Devices_1.pdf
Source snippet
Stronger FDA Approval Standards for AI Medical...by E Wu · 2022 · Cited by 5 — We show performance degradation—and potential demographic...
Source: pew.org
Title: fda review can limit bias risks in medical devices using artificial intelligence
Link: https://www.pew.org/en/research-and-analysis/articles/2021/10/07/fda-review-can-limit-bias-risks-in-medical-devices-using-artificial-intelligence
Source snippet
FDA Review Can Limit Bias Risks in Medical Devices...7 Oct 2021 — The algorithms that underpin AI software may have hidden biases that l...