Automation bias

Introduction

AI can help doctors catch things they might otherwise miss. But the same systems that act as a useful “second set of eyes” can also create a dangerous psychological effect: automation bias. This happens when clinicians begin trusting machine suggestions too readily, especially under time pressure, fatigue, or information overload. In practice, the risk is not usually that doctors stop thinking entirely. It is subtler. A confident AI recommendation can quietly steer attention, suppress doubt, or discourage a second look.

Overview image for Automation bias That matters because the optimistic case for AI in medicine depends on human judgement remaining active. AI-assisted diagnosis could eventually expand access to expert-level screening and reduce avoidable errors across entire populations. But if clinicians start treating AI outputs as authoritative rather than advisory, the technology can amplify mistakes instead of catching them. Research in radiology, mammography, pathology, and clinical decision support increasingly shows that human-AI collaboration is not automatically safe merely because a human remains “in the loop”. RSNA Publications Online [JAMA Network]jamanetwork.comJAMA NetworkRisk of Harm From AI-Driven Clinical Decision Supportby R Khera · 2023 · Cited by 179 — Automation Bias and Assistive AI: Ris…

Why clinicians may defer to AI

Automation bias is not unique to medicine. Pilots, financial analysts, and military operators have all shown a tendency to trust automated systems even when those systems are wrong. Healthcare adds extra pressures that can make the effect stronger.

Modern clinicians often work in environments defined by cognitive overload: overflowing inboxes, scan backlogs, alert fatigue, and constant interruptions. In that context, AI recommendations can feel less like optional advice and more like a cognitive shortcut. If an algorithm has been correct hundreds of times before, it becomes psychologically harder to challenge it on the thousand-and-first case.

Researchers often distinguish between two related failures:

Errors of omission: a clinician misses a real problem because the AI failed to flag it.
Errors of commission: a clinician accepts an incorrect AI suggestion despite contradictory evidence.

The second category is particularly important because it shows that AI can actively distort judgement, not merely fail to help. [ScienceDirect]sciencedirect.comScienceDirectExploring the risks of automation bias in healthcare…by M Abdelwanis · 2024 · Cited by 149 — This study conducts an in-de…

The problem becomes sharper when AI systems present outputs with high confidence, simplified visual markers, or authoritative language. A highlighted region on a scan can strongly anchor a radiologist’s attention. Once attention narrows around the AI’s suggestion, alternative interpretations may receive less scrutiny. Some studies describe this as a combination of automation bias and anchoring bias: the machine’s first answer becomes the mental reference point for the human reviewer. [Nature]nature.comNatureThe impact of AI suggestions on radiologists' decisionsby MH Rezazade Mehrizi · 2023 · Cited by 46 — We examine the effect of corre…

There is also a social dimension. In medicine, disagreeing with software may eventually require justification. If hospitals begin measuring compliance with AI-assisted workflows, clinicians could feel pressure to follow the machine even when uncertain. The danger is not only technical failure but institutional drift toward passive oversight.

Automation bias illustration 1

What mammography studies reveal

Mammography has become one of the clearest real-world laboratories for studying automation bias because breast cancer screening already depends on subtle image interpretation under heavy workload conditions.

A widely discussed 2023 study in Radiology examined how radiologists responded when AI systems intentionally provided incorrect BI-RADS assessments, the standard classification system used in breast imaging. Researchers found that radiologists across all experience levels became susceptible to automation bias when exposed to erroneous AI suggestions. Diagnostic accuracy dropped when clinicians followed incorrect machine guidance. rsna.org 3RSNA Publications Online [PubMed One striking finding was that even highly experienced radiologists were affected. Expertise reduced the problem somewhat]kslaw.comfda updates general wellness and clinical decision support guidance documentsKing & SpaldingFDA Updates General Wellness and Clinical Decision…Jan 9, 2026 — The updated guidance includes a discussion of automati…, but did not eliminate it. That matters because it weakens the comforting assumption that only junior staff are vulnerable to AI overreliance.

The mechanism was not simply laziness. The AI altered how radiologists interpreted ambiguous cases. In some situations, clinicians revised initially correct judgements after seeing incorrect AI output. In effect, the machine changed the human answer from right to wrong. [RSNA Publications Online]pubs.rsna.orgRSNA Publications OnlineAutomation Bias in Mammography: The Impact of Artificial…by T Dratsch · 2023 · Cited by 265 — The results show…

This creates a tension at the centre of medical AI deployment. The same systems can improve average performance overall while still introducing new categories of error. Large trials of AI-assisted mammography, including national-scale deployments in Europe, suggest that AI can reduce workload and help identify cancers earlier. [Nature]nature.comNatureNationwide real-world implementation of AI for cancer…by N Eisemann · 2025 · Cited by 168 — AI-supported double reading was asso… But those gains do not automatically mean the human-AI partnership is psychologically robust.

Researchers in radiology increasingly argue that the critical safety question is no longer just “How accurate is the model?” but “How does the model change clinician behaviour?” [Nature]nature.comNatureHeterogeneity and predictors of the effects of AI assistance…by F Yu · 2024 · Cited by 185 — This large-scale study examined the…

The risk grows under pressure and routine

Automation bias tends to worsen in exactly the situations where AI systems are most attractive: high-volume routine work.

In pathology experiments, researchers found that AI support improved average performance overall while still producing measurable automation bias. In some cases, pathologists changed correct answers to incorrect ones after receiving flawed AI advice. Time pressure appeared to intensify the severity of these effects. [arXiv]arxiv.orgSource details in endnotes.

This reflects a broader human tendency. When mental bandwidth is limited, people rely more heavily on heuristics and external cues. AI recommendations become cognitively tempting because they reduce uncertainty and speed decisions.

Ironically, very successful AI can increase this risk. The more often a system is correct, the harder it becomes to maintain healthy scepticism. Humans are poor at sustaining calibrated distrust toward tools that usually work. A radiologist who sees hundreds of useful alerts may eventually stop independently verifying borderline cases.

That dynamic may become more important as generative AI systems move beyond image flagging into report drafting and clinical summarisation. Some researchers warn that automatically generated radiology reports could create even stronger automation bias because clinicians may begin editing machine-written conclusions rather than reasoning from raw evidence themselves. [pc.kjronline.org]pc.kjronline.orgThis is because theseResponse to Comments on “Artificial Intelligence-Driven Drafting of…Unlike traditional computer-aided detection systems, generative AI…

The long-term concern is skill erosion. If clinicians increasingly supervise AI instead of performing full independent analysis, diagnostic expertise could weaken over time. Medicine could gradually produce professionals who are highly competent at managing AI systems but less practised at detecting subtle errors without algorithmic support.

Automation bias illustration 2

Why “human in the loop” is not enough

Public discussion often treats human oversight as a simple safety guarantee. But the evidence suggests that merely keeping a clinician nominally involved does not automatically prevent automation bias.

A doctor who clicks “approve” after briefly reviewing an AI recommendation may technically remain responsible while functionally acting as a rubber stamp. Regulators have begun recognising this distinction. Recent US Food and Drug Administration guidance on clinical decision support explicitly discusses automation bias as a safety risk and emphasises that clinicians must be able to independently review the basis for AI recommendations. [U.S. Food and Drug Administration]fda.govU.S. Food and Drug AdministrationWe Regulate Food Drugs, Medical Devices, Radiation-Emitting Products, Vaccines, Blood, and Biologics, An… [King & Spalding]kslaw.comfda updates general wellness and clinical decision support guidance documentsKing & SpaldingFDA Updates General Wellness and Clinical Decision…Jan 9, 2026 — The updated guidance includes a discussion of automati…

This reflects a broader shift in thinking about AI safety in medicine. Earlier debates focused mainly on model accuracy. Increasingly, attention is moving toward human factors: interface design, workflow structure, explainability, alert timing, and the psychology of trust.

An AI system can be statistically impressive yet operationally unsafe if it subtly degrades clinician vigilance. Conversely, a less powerful model may produce better outcomes if it encourages active engagement rather than passive acceptance.

That distinction matters for the wider AI bloom argument. If advanced AI is to expand human capability safely, institutions will need to preserve meaningful human agency rather than merely symbolic oversight. The challenge is not only building intelligent systems, but designing relationships between humans and machines that keep human judgement alive.

Design habits that keep human judgement active

Researchers and regulators are increasingly converging on a practical insight: safe medical AI depends as much on workflow design as on algorithm quality.

Several design approaches appear promising.

Delayed AI exposure

One approach is requiring clinicians to form an initial judgement before seeing the AI recommendation. This preserves independent reasoning and reduces anchoring effects.

Some studies suggest that when radiologists first inspect scans unaided and only later consult AI support, automation bias weakens. The clinician remains the primary interpreter rather than becoming a verifier of machine output. [Nature]nature.comd the FDA and global regulators to shift toward governance frameworks…Read more…

Automation bias illustration 3

Showing uncertainty rather than false certainty

AI systems often present outputs in ways that imply more confidence than the underlying evidence justifies. Binary labels such as “normal” or “high risk” can encourage over-trust.

Designers increasingly argue for interfaces that expose uncertainty ranges, competing possibilities, or reasons for low confidence. A cautious AI may paradoxically produce safer human behaviour than an overly polished one.

Keeping disagreement visible

Human teams often improve decisions through constructive disagreement. AI systems can suppress that process if clinicians feel socially or institutionally pressured to align with the algorithm.

Some researchers advocate workflows that explicitly record human-AI disagreement rather than treating disagreement as failure. In this model, conflict becomes diagnostically useful rather than administratively inconvenient.

Training clinicians to distrust appropriately

Medical training may need to change alongside the technology. Clinicians increasingly require not only diagnostic skills but also “AI literacy”: understanding where models fail, how datasets shape outputs, and how cognitive bias affects human-AI interaction.

Importantly, the goal is not hostility toward AI. Excessive distrust can also reduce performance. The real challenge is calibrated trust: using AI as a cognitive amplifier without surrendering independent judgement. [PMC]pmc.ncbi.nlm.nih.govPMCBias in artificial intelligence for medical imagingAlgorithmic aversion refers to a…Read more…

The deeper lesson for AI-enabled medicine

Automation bias reveals something important about the broader future of AI in healthcare. The central challenge is not simply whether machines can outperform humans on benchmarks. It is whether human institutions can absorb increasingly capable systems without weakening the human capacities those systems are meant to support.

The optimistic vision of AI-assisted medicine remains plausible. AI could help expand screening coverage, reduce diagnostic backlogs, accelerate scientific discovery, and make expert-level support available far beyond elite hospitals. In that sense, medical AI may become an early example of AI-enabled abundance: more clinical attention, more preventive care, and faster detection at population scale.

But the mammography and radiology evidence shows that this future is not achieved automatically by adding AI into workflows. A second clinical eye only helps if the first human eye stays meaningfully engaged. RSNA Publications Online [JAMA Network]jamanetwork.comJAMA NetworkRisk of Harm From AI-Driven Clinical Decision Supportby R Khera · 2023 · Cited by 179 — Automation Bias and Assistive AI: Ris…

Endnotes

Source: pubs.rsna.org
Link: https://pubs.rsna.org/doi/abs/10.1148/radiol.222176
Source snippet
RSNA Publications OnlineAutomation Bias in Mammography: The Impact of Artificial...by T Dratsch · 2023 · Cited by 265 — The results show...
Source: sciencedirect.com
Link: https://www.sciencedirect.com/science/article/pii/S2666449624000410
Source snippet
ScienceDirectExploring the risks of automation bias in healthcare...by M Abdelwanis · 2024 · Cited by 149 — This study conducts an in-de...
Source: nature.com
Link: https://www.nature.com/articles/s41598-023-36435-3
Source snippet
NatureThe impact of AI suggestions on radiologists' decisionsby MH Rezazade Mehrizi · 2023 · Cited by 46 — We examine the effect of corre...
Source: arxiv.org
Link: https://arxiv.org/abs/2603.11821
Source: rsna.org
Title: ai bias may impair accuracy
Link: https://www.rsna.org/news/2023/may/ai-bias-may-impair-accuracy
Source snippet
AI Bias May Impair Radiologist Accuracy on Mammogram2 May 2023 — Several studies have shown that the introduction of computer-aided detec...

Published: May 2023
Source: rsna.org
Link: https://www.rsna.org/media/press/i/2432
Source snippet
AI Bias May Impair Radiologist Accuracy on MammogramMay 2, 2023 — The findings support the theory that use of AI assistance in mammograph...

Published: May 2, 2023
Source: nature.com
Link: https://www.nature.com/articles/s41591-024-03408-6
Source snippet
NatureNationwide real-world implementation of AI for cancer...by N Eisemann · 2025 · Cited by 168 — AI-supported double reading was asso...
Source: nature.com
Link: https://www.nature.com/articles/s41591-024-02850-w
Source snippet
NatureHeterogeneity and predictors of the effects of AI assistance...by F Yu · 2024 · Cited by 185 — This large-scale study examined the...
Source: kjronline.org
Link: https://www.kjronline.org/DOIx.php?id=10.3348%2Fkjr.2025.0071
Source snippet
Crucial Role of Understanding in Human-Artificial...by SH Park · 2025 · Cited by 16 — These findings highlight two critical issues: firs...
Source: arxiv.org
Link: https://arxiv.org/abs/2411.00998
Source snippet
arXivAutomation Bias in AI-Assisted Medical Decision-Making under Time Pressure in Computational PathologyNovember 1, 2024...

Published: November 1, 2024
Source: pc.kjronline.org
Title: This is because these
Link: https://pc.kjronline.org/DOIx.php?id=10.3348%2Fkjr.2026.0028
Source snippet
Response to Comments on “Artificial Intelligence-Driven Drafting of...Unlike traditional computer-aided detection systems, generative AI...
Source: fda.gov
Link: https://www.fda.gov/media/109618/download
Source snippet
Food and Drug AdministrationClinical Decision Support Software - Guidance for Industry...Jan 29, 2026 — Automation bias is the propensit...
Source: pmc.ncbi.nlm.nih.gov
Title: PMCBias in artificial intelligence for medical imaging
Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC11880872/
Source snippet
Algorithmic aversion refers to a...Read more...
Source: rsna.org
Title: ai influences diagnostic decisions
Link: https://www.rsna.org/news/2024/november/ai-influences-diagnostic-decisions
Source snippet
AI could be a double-edged sword because it risks over-reliance or automation bias. “When we rely too much on whatever the computer tells...
Source: nature.com
Link: https://www.nature.com/articles/s41746-026-02561-1
Source snippet
d the FDA and global regulators to shift toward governance frameworks...Read more...
Source: nature.com
Link: https://www.nature.com/articles/s41746-024-01270-x
Source snippet
Regulatory oversight is critical in...
Source: pubs.rsna.org
Link: https://pubs.rsna.org/doi/abs/10.1148/radiol.230770
Source snippet
more...
Source: rsna.org
Link: https://www.rsna.org/media/press/i/2552
Source snippet
ncorrect AI Advice Influences Diagnostic DecisionsNov 19, 2024 — — When making diagnostic decisions, radiologists and other physicians ma...
Source: sciencedirect.com
Title: •. Overreliance on errant AI findings
Link: https://www.sciencedirect.com/science/article/abs/pii/S0899707126000379
Source snippet
Automation bias and overconfidence in artificial...by JL Mezrich · 2026 · Cited by 1 — As AI becomes prevalent, radiologists may succumb...
Source: jamanetwork.com
Link: https://jamanetwork.com/journals/jama/article-abstract/2812931
Source snippet
JAMA NetworkRisk of Harm From AI-Driven Clinical Decision Supportby R Khera · 2023 · Cited by 179 — Automation Bias and Assistive AI: Ris...
Source: kslaw.com
Title: fda updates general wellness and clinical decision support guidance documents
Link: https://www.kslaw.com/news-and-insights/fda-updates-general-wellness-and-clinical-decision-support-guidance-documents
Source snippet
King & SpaldingFDA Updates General Wellness and Clinical Decision...Jan 9, 2026 — The updated guidance includes a discussion of automati...
Source: pmc.ncbi.nlm.nih.gov
Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC11255031/
Source snippet
PMC - NIHby H Al-Bazzaz · 2024 · Cited by 26 — The shift in study radiologist operating point caused by concurrent AI decision support de...
Source: Wikipedia
Link: https://en.wikipedia.org/wiki/Automation
Source snippet
AutomationAutomation describes a wide range of technologies that reduce human intervention in processes, mainly by predetermining deci...
Source: fda.gov
Link: https://www.fda.gov/
Source snippet
U.S. Food and Drug AdministrationWe Regulate Food Drugs, Medical Devices, Radiation-Emitting Products, Vaccines, Blood, and Biologics, An...

Additional References

Source: oecd.ai
Link: https://oecd.ai/en/incidents/2023-05-02-9b21
Source snippet
AI-Induced Automation Bias Reduces Mammogram...A study found that radiologists using AI-based decision support for mammography are prone...
Source: ropesgray.com
Link: https://www.ropesgray.com/en/insights/alerts/2026/01/fda-adapts-with-the-times-on-digital-health-updated-guidances-on-general-wellness-products
Source snippet
FDA Adapts with the Times on Digital Health: Updated...Less clear is how concerned FDA will remain about “automation bias.” As noted, au...
Source: linkedin.com
Link: https://www.linkedin.com/posts/alex-cadotte-439bb0171_one-of-the-sections-now-heavily-edited-by-activity-7415124448812539906-fpGO
Source snippet
FDA Guidance on Automation Bias in AI Medical DevicesJan 8, 2026 — One of the sections now heavily edited by the new Clinical Decision Su...
Source: linkedin.com
Link: https://www.linkedin.com/posts/shannonhoste_clinical-decision-support-software-guidance-activity-7427409582370758658-ubHa
Source: annualreviews.org
Link: https://www.annualreviews.org/content/journals/10.1146/annurev-biodatasci-103123-095824?TRACK=RSS
Source snippet
Weissman GE. 2020.. FDA regulation of predictive clinical decision-support tools: What does it mean for hospitals?...
Source: aa.com.tr
Title: ai in healthcare may offer great potential but data bias risks remain
Link: https://aa.com.tr/en/life/ai-in-healthcare-may-offer-great-potential-but-data-bias-risks-remain/3837777
Source snippet
AI in healthcare may offer great potential, but data bias...Feb 23, 2026 — Medical applications of AI could amplify diagnostic errors if...
Source: dscience.com
Title: artificial intelligence human factors engineering fda updates
Link: https://dscience.com/blog/artificial-intelligence-human-factors-engineering-fda-updates
Source snippet
What FDA's AI-Enabled Device Guidance Means for Human...Dec 9, 2025 — FDA emphasizes a two-pronged approach to validating AI-enabled dev...
Source: news-medical.net
Title: AI explanations in radiology can lead to over reliance.aspx
Link: https://www.news-medical.net/news/20241119/AI-explanations-in-radiology-can-lead-to-over-reliance.aspx
Source snippet
AI could be a double-edged sword because it risks over-reliance or automation bias. "When we rely too much on whatever the computer tells...
Source: spie.org
Link: https://spie.org/medical-imaging/presentation/Automation-bias-and-timing-of-artificial-intelligence-decision-support-in/13928-29
Source snippet
Automation bias and timing of artificial intelligence decision...All radiologists preferred to use AI support, and seven preferred it to...
Source: axisimagingnews.com
Title: study explores radiologists reliance on ai in diagnostics
Link: https://axisimagingnews.com/market-trends/cloud-computing/machine-learning-ai/study-explores-radiologists-reliance-on-ai-in-diagnostics
Source snippet
Study Explores Radiologists' Reliance on AI in DiagnosticsNov 20, 2024 — Summary: A study in Radiology found that while AI assistance imp...