When AI Healthcare Gets Need Wrong

Introduction

AI could help create a future of better healthcare, longer healthy lives and wider access to medical expertise. But the same systems that promise faster diagnosis and more efficient care can also miss people whose needs are poorly represented by the data or measurements they use. Healthcare algorithms do not only make mistakes because they are inaccurate; they can fail because they optimise the wrong target. A system may appear efficient while quietly overlooking groups who have historically received less care, less spending or less attention.[NCBI]ncbi.nlm.nih.govSummary - Impact of Healthcare Algorithms on Racial and Ethnic Disparities in Health and Healthcare - NCBI Bookshelf…

The clearest example is a widely used US healthcare algorithm designed to identify patients who would benefit from extra support. The algorithm did not use race as an input, yet it underestimated the needs of many Black patients because it used healthcare spending as a stand-in for illness. Since unequal access and treatment patterns meant that similar levels of illness often produced different spending levels, the algorithm learned a distorted picture of need. The case shows a central challenge for AI-enabled healthcare: improving medicine requires not only better prediction, but better definitions of what matters.[ScienceOpen]scienceopen.comOpen source on scienceopen.com.

For an AI-enabled human bloom, this distinction is fundamental. Advanced systems could eventually accelerate medical discovery, personalise treatment and help extend healthy human life. But those gains depend on whether AI systems recognise human needs broadly rather than simply reproducing the patterns left behind by existing healthcare systems.

How spending became a proxy for health need

Many healthcare algorithms work by predicting a measurable outcome. The difficulty is that the easiest outcome to measure is not always the outcome society actually cares about.

In the case studied by Ziad Obermeyer and colleagues, researchers examined an algorithm used to identify patients likely to benefit from additional care management programmes. The system predicted future healthcare costs, assuming that higher expected costs indicated greater medical need. This was a practical choice because costs were recorded consistently in healthcare data. However, cost was only an indirect measure of illness.[ScienceOpen]scienceopen.comOpen source on scienceopen.com.

The problem was that healthcare spending reflects more than disease severity. It also reflects access to doctors, insurance coverage, patterns of diagnosis, trust in healthcare institutions and whether patients receive early treatment. When those factors differ between groups, spending becomes a biased measure of underlying health needs.[NCBI]ncbi.nlm.nih.govSummary - Impact of Healthcare Algorithms on Racial and Ethnic Disparities in Health and Healthcare - NCBI Bookshelf…

The algorithm was therefore successful at its narrow task: it predicted future costs reasonably well. But the healthcare decision being made was different. The real question was not “Who will cost the healthcare system the most?” It was “Who is most likely to need additional medical support?” Those two questions overlap, but they are not identical.

This is a common mechanism behind algorithmic unfairness in healthcare. A model can be statistically accurate while still producing unequal outcomes because the target it learns is incomplete. Researchers in medical AI describe similar problems arising from biased data collection, unequal clinical workflows and differences in how diseases are labelled or measured.[Nature]nature.comAlgorithmic fairness in artificial intelligence for medicine and healthcare | Nature Biomedical EngineeringJune 28, 2023…Published: June 28, 2023

What the algorithm missed for Black patients

The consequences of the cost-based approach were substantial. Obermeyer and colleagues found that, at the same risk score, Black patients were significantly sicker than White patients. When researchers changed the target from healthcare costs to actual health conditions, the racial disparity was greatly reduced. The study estimated that correcting the approach would increase the proportion of Black patients identified for additional support from 17.7% to 46.5%.[ScienceOpen]scienceopen.comOpen source on scienceopen.com.

The finding was important because it showed that algorithmic bias does not require explicit discrimination. The algorithm did not need a race variable to produce unequal results. Historical differences in healthcare access and treatment patterns were already embedded in the data used for training. The model inherited those patterns through a seemingly neutral measurement.[eScholarship]escholarship.orgOpen source on escholarship.org.

The lesson extends beyond one healthcare programme. Medical AI systems increasingly assist with diagnosis, prioritisation and treatment decisions, meaning choices about data and objectives can affect who receives attention. A review by the US Agency for Healthcare Research and Quality examined evidence from healthcare algorithms and found that these systems can influence disparities in access, quality of care and health outcomes, although the effects vary depending on the algorithm and setting.[NCBI]ncbi.nlm.nih.govNCBI BookshelfDecember 1, 2023…Published: December 1, 2023

Medical imaging provides another example of how unequal needs can be missed. A study in Nature Medicine examined AI systems for analysing chest X-rays and found that several models showed higher underdiagnosis rates among historically underserved groups, including Black patients, women and people with lower socioeconomic status. The researchers warned that underdiagnosis is especially concerning because it can delay treatment for people who already face barriers to care.[Nature]nature.comUnderdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations | Nature…

These examples reveal a broader pattern: the groups most likely to be harmed by healthcare algorithms are often those whose experiences are least fully captured by existing datasets. A model trained on past healthcare patterns may learn what the system has historically noticed, rather than what patients actually need.

Building fairer medical AI decisions

Fairer healthcare AI does not come from simply removing sensitive information such as race from datasets. The central issue is often the relationship between inputs, targets and outcomes. A system can ignore race while still reflecting racial inequalities through variables such as spending, location, previous treatment or healthcare access.[NCBI]ncbi.nlm.nih.govSummary - Impact of Healthcare Algorithms on Racial and Ethnic Disparities in Health and Healthcare - NCBI Bookshelf…

A more reliable approach begins with asking the right question before building the model. Developers and healthcare organisations need to decide whether the prediction target represents the clinical goal or merely a convenient measurement. Predicting cost, hospital visits or administrative outcomes may be useful, but those measures should not automatically be treated as direct measures of patient need.

Several practices are increasingly viewed as important safeguards:

• Measure outcomes across groups: Healthcare AI systems should be tested separately across relevant populations rather than judged only by average performance. A model that performs well overall may still fail badly for specific communities.[Nature]nature.comAlgorithmic fairness in artificial intelligence for medicine and healthcare | Nature Biomedical EngineeringJune 28, 2023…Published: June 28, 2023
• Use representative data: Training and evaluation datasets should reflect the people who will actually use the system. Regulators and health organisations increasingly emphasise documenting which populations are represented and where gaps remain.[U.S. Food and Drug Administration]fda.govOpen source on fda.gov.
• Monitor real-world performance: Medical AI can behave differently after deployment because hospitals, patients and environments differ from the original training data. Ongoing review is needed to detect new forms of unequal performance.[U.S. Food and Drug Administration]fda.govOpen source on fda.gov.
• Keep human judgement involved: Algorithms should support clinical decisions rather than replace responsibility for understanding patients’ circumstances. A technically strong prediction cannot compensate for a poorly chosen objective.

The UK’s independent review of equity in medical devices highlighted a similar concern: some medical technologies have shown unfair performance differences linked to factors such as ethnicity, sex or socioeconomic circumstances, often because of design choices or unrepresentative testing rather than deliberate intent.[GOV.UK]GOV.UKEquity in medical devices: independent reviewfinal report - GOV.UKMarch 11, 2024…Published: March 11, 2024

Why this matters for the future of AI-enabled healthcare

The optimistic case for AI in medicine depends on solving exactly this kind of problem. If advanced AI systems can discover treatments, analyse medical knowledge and provide personalised care, they could help reduce suffering on a scale beyond what current healthcare systems can achieve. But abundance in healthcare is not only about producing more treatments; it is about ensuring that the right people can benefit from them.

Healthcare algorithms that miss unequal needs show why technical capability alone is not enough. The challenge is aligning powerful tools with human goals: health, dignity and fair access to care. A future where AI helps humanity flourish will require systems that recognise hidden gaps rather than treating existing patterns as the definition of what people deserve.

Amazon book picks

Further Reading

Books and field guides related to When AI Healthcare Gets Need Wrong. Use these as the next step if you want deeper reading beyond the article.

Book

The AI Revolution in Medicine

By Peter Lee, Carey Goldberg et al.

AI is about to transform medicine. Here's what you need to know right now. ''The development of AI is as fundamental as the creation of t...

See on Amazon

Book

Weapons of Math Destruction

By Cathy O'Neil

'A manual for the 21st-century citizen... accessible, refreshingly critical, relevant and urgent' - Financial Times 'Fascinating and deep...

See on Amazon

Book

Algorithms of Oppression

By Safiya Umoja Noble

Acknowledgments -- Introduction: the power of algorithms -- A society, searching -- Searching for Black girls -- Searching for people and...

See on Amazon

Book

Unwell Women

By Elinor Cleghorn

Medicine carries the burden of its own troubling history. Over centuries, women's bodies have been demonised and demeaned until we feared...

See on Amazon

Browse more on Amazon:The AI Revolution in MedicineWeapons of Math DestructionAlgorithms of Oppression

As an Amazon Associate I earn from qualifying purchases.

eBay marketplace picks

Marketplace Samples

Live-tested eBay searches with available results related to this page.

Shop location

UsingUSA USA USAUKCanadaAustraliaIreland

Selected frommedical technology poster oneBay.co.uk.

Current eBay listing

Medical Inventions Poster - Medical Technology Wall Art 12X16/16X20/18X24/24X36

GBP 16.99 | Free shippingNewSeller: the wall art store (99.5% positive)Ships from: IE

View listing on eBay

Current eBay listing

Medical Technology Intern T Shirt - Framed Wall Art Poster Canvas Print Picture

GBP 8.99 | Shipping GBP 4.95NewSeller: nobleart (98.7% positive)Ships from: GB

View listing on eBay

Current eBay listing

virus scan technology world medical Framed Wall Art Poster Canvas Print Picture

GBP 14.99 | Shipping GBP 4.95NewSeller: silkframe (99.7% positive)Ships from: GB

View listing on eBay

Browse more oneBay.co.uk

Example items shown for inspiration; availability and pricing can change. Branchoria may earn a commission if you purchase through outbound eBay links.

Endnotes

1. Source: ncbi.nlm.nih.gov
Link:https://www.ncbi.nlm.nih.gov/books/NBK598808/

2. Source: scienceopen.com
Link:https://www.scienceopen.com/document?vid=34b5b9f0-d108-4873-b244-c0f785b635a8

3. Source: escholarship.org
Link:https://escholarship.org/uc/item/6h92v832

4. Source: nature.com
Link:https://www.nature.com/articles/s41551-023-01056-8

Published: June 28, 2023

5. Source: ncbi.nlm.nih.gov
Link:https://www.ncbi.nlm.nih.gov/books/NBK598802/

Published: December 1, 2023

6. Source: nature.com
Link:https://www.nature.com/articles/s41591-021-01595-0

7. Source: GOV.UK
Title: Equity in medical devices: independent review
Link:https://www.gov.uk/government/publications/equity-in-medical-devices-independent-review-final-report

Published: March 11, 2024

8. Source: GOV.UK
Title: Equity in medical devices: independent review
Link:https://www.gov.uk/government/publications/equity-in-medical-devices-independent-review-final-report/equity-in-medical-devices-independent-review-summary-report

9. Source: nature.com
Title: Disparate privacy risks from medical AI | Nature
Link:https://www.nature.com/articles/s41586-026-10688-0

10. Source: nature.com
Link:https://www.nature.com/articles/s41746-026-02723-1

11. Source: nature.com
Link:https://www.nature.com/articles/s41591-025-03626-6

12. Source: nature.com
Link:https://www.nature.com/articles/s41746-025-01503-7

13. Source: nature.com
Link:https://www.nature.com/articles/s41591-024-03258-2

14. Source: nature.com
Link:https://www.nature.com/articles/s41591-024-03113-4

15. Source: GOV.UK
Link:https://www.gov.uk/government/publications/machine-learning-medical-devices-transparency-principles/transparency-for-machine-learning-enabled-medical-devices-guiding-principles

16. Source: nature.com
Link:https://www.nature.com/articles/s41591-024-02838-6

17. Source: GOV.UK
Title: www.gov.uk Equity in medical devices: independent review
Link:https://www.gov.uk/government/publications/equity-in-medical-devices-independent-review-final-report/equity-in-medical-devices-independent-review-quick-read

18. Source: GOV.UK
Link:https://www.gov.uk/government/publications/government-response-to-the-report-of-the-equity-in-medical-devices-independent-review

19. Source: GOV.UK
Title: www.gov.uk New action to tackle ethnic and other biases in medical devices
Link:https://www.gov.uk/government/news/new-action-to-tackle-ethnic-and-other-biases-in-medical-devices

20. Source: GOV.UK
Title: www.gov.uk Medical devices: software and artificial intelligence (AI)
Link:https://www.gov.uk/government/publications/software-and-artificial-intelligence-ai-as-a-medical-device

21. Source: GOV.UK
Link:https://www.gov.uk/government/publications/good-machine-learning-practice-for-medical-device-[development

22. Source: fda.gov
Link:https://www.fda.gov/medical-devices/software-medical-device-samd/transparency-machine-learning-enabled-medical-devices-guiding-principles

23. Source: fda.gov
Link:https://www.fda.gov/news-events/press-announcements/fda-issues-comprehensive-draft-guidance-developers-artificial-intelligence-enabled-medical-devices

24. Source: pubmed.ncbi.nlm.nih.gov
Link:https://pubmed.ncbi.nlm.nih.gov/39695057/

25. Source: pmc.ncbi.nlm.nih.gov
Link:https://pmc.ncbi.nlm.nih.gov/articles/PMC12934203/

26. Source: fda.gov
Link:https://www.fda.gov/medical-devices/medical-devices-news-and-events/cdrh-issues-guiding-principles-transparency-machine-learning-enabled-medical-devices

27. Source: pubmed.ncbi.nlm.nih.gov
Link:https://pubmed.ncbi.nlm.nih.gov/38467001/

28. Source: pubmed.ncbi.nlm.nih.gov
Link:https://pubmed.ncbi.nlm.nih.gov/38100101/

29. Source: ncbi.nlm.nih.gov
Link:https://www.ncbi.nlm.nih.gov/books/NBK598796/

30. Source: pmc.ncbi.nlm.nih.gov
Link:https://pmc.ncbi.nlm.nih.gov/articles/PMC10668606/

31. Source: fda.gov
Link:https://www.fda.gov/medical-devices/medical-devices-news-and-events/cdrh-issues-guiding-principles-predetermined-change-control-plans-machine-learning-enabled-medical

32. Source: pubmed.ncbi.nlm.nih.gov
Link:https://pubmed.ncbi.nlm.nih.gov/31649194/

33. Source: ncbi.nlm.nih.gov
Link:https://www.ncbi.nlm.nih.gov/books/NBK598794/

34. Source: ncbi.nlm.nih.gov
Link:https://www.ncbi.nlm.nih.gov/sites/books/NBK598794/

35. Source: pubmed.ncbi.nlm.nih.gov
Link:https://pubmed.ncbi.nlm.nih.gov/37782868/

36. Source: fda.gov
Link:https://www.fda.gov/regulatory-information/search-fda-guidance-documents/evaluation-and-reporting-age-race-and-ethnicity-specific-data-medical-device-clinical-studies

Additional References

37. Source: youtube.com
Title: Dissecting Racial Bias in an Algorithm that Guides Health Decisions for Millions
Link:https://www.youtube.com/watch?v=y6eo0FZIqjk

38. Source: youtube.com
Title: Dissecting Algorithmic Bias | Ziad Obermeyer | AI FOR GOOD DISCOVERY
Link:https://www.youtube.com/watch?v=U5MlyFsMi-E

39. Source: youtube.com
Title: Ziad Obermeyer on tackling ‘data bottleneck’ to combat AI bias
Link:https://www.youtube.com/watch?v=xWLGch3g3mA

40. Source: ahrq.gov
Link:https://www.ahrq.gov/news/newsroom/press-releases/guiding-principles.html

41. Source: youtube.com
Title: Keynote Presentation: Dissecting Algorithmic Bias
Link:https://www.youtube.com/watch?v=JfKYO1W4uuA

42. Source: doi.org
Link:https://doi.org/10.1038/s41551-023-01056-8

43. Source: bmj.com
Link:https://www.bmj.com/content/388/bmj-2024-081554

44. Source: jamanetwork.com
Link:https://jamanetwork.com/journals/jamanetworkopen/fullarticle/2812958

45. Source: nice.org.uk
Link:https://www.nice.org.uk/corporate/ecd11/chapter/our-position-on-the-use-of-ai-in-evidence-generation-and-reporting

46. Source: nice.org.uk
Link:https://www.nice.org.uk/about-us/our-principles