Can Officials Trust AI They Cannot Fully Explain?

Introduction

Public health agencies should not automatically trust black-box outbreak models, but neither should they reject them simply because they are difficult to explain. The practical question is not whether an AI system is perfectly interpretable. It is whether officials can understand enough about its strengths, limits, uncertainty, and failure modes to use it responsibly when lives and resources are at stake.

This matters because outbreak forecasting is becoming more ambitious. Deep learning systems can detect subtle patterns across mobility data, hospital records, weather signals, genomic information, news reports, and online behaviour. In some cases they have improved short-term forecasting and provided earlier warning signals than traditional surveillance alone. Yet many of the most powerful systems operate as statistical black boxes whose internal reasoning is difficult even for their creators to interpret. Public-health leaders therefore face a dilemma: ignore potentially valuable warnings, or act on forecasts they cannot fully explain.[CDC]cdc.govCenter for Forecasting and Outbreak Analytics | CFACFA uses advanced analytic approaches, like forecasting and modeling, to drive effe…[Public Health AI Handbook]publichealthaihandbook.comPublic Health AI Handbook Epidemic Forecasting with AICOVID-19 experiments, PandemicLLM reported better 1- to 3-week forecasting performance than several CDC COVID-19 Forecast Hub baselines…

Within the broader discussion of AI-enabled civilisational resilience, this debate sits at the boundary between capability and governance. Better forecasting could help societies detect and contain outbreaks earlier, reducing mortality and economic disruption. But if agencies become dependent on opaque systems they do not understand, forecasting tools can create new risks alongside new capabilities.

Why Deep Learning Forecasts Are Hard to Interpret

Traditional epidemiological models often expose their assumptions. A researcher can usually explain how infection rates, contact patterns, immunity, or population movement contribute to a forecast. Deep learning models work differently.

Modern neural networks may absorb enormous quantities of information and identify correlations that no human analyst would notice. Their forecasts emerge from millions or billions of internal parameters rather than a transparent chain of reasoning. Even when the prediction is accurate, it may be difficult to answer a simple question: why did the model expect a surge in cases three weeks from now?[ScienceDirect]sciencedirect.comMedical artificial intelligence and the black box problemby H Xu · 2024 · Cited by 199 — In this study, we focus on the pote…[PMC]pmc.ncbi.nlm.nih.govUnbox the black-box for the medical explainable AI via multi…by G Yang · 2022 · Cited by 887 — Explainable Artificial Intelligence…

This opacity creates several distinct problems:

• Verification becomes harder. Officials cannot easily check whether the model is relying on sensible epidemiological signals or accidental correlations.
• Errors become harder to diagnose. When forecasts fail, it may be unclear whether the problem came from poor data, changing disease dynamics, or flaws in the model itself.
• Novel situations expose weaknesses. Models trained on historical outbreaks may struggle when a pathogen behaves differently from anything in their training data.
• Public communication becomes more difficult. Leaders may have to justify costly interventions without being able to explain exactly how a forecast was generated.[ScienceDirect]sciencedirect.comMedical artificial intelligence and the black box problemby H Xu · 2024 · Cited by 199 — In this study, we focus on the pote…[PHG Foundation]phgfoundation.orgBlack box medicine and transparencyThis last section outlines two cases that illustrate the importance of interpretability in machine lea…

The problem is not unique to outbreak forecasting. Similar debates have emerged across medicine, where highly accurate systems sometimes remain difficult to interpret. Researchers increasingly describe trust, accountability, and explainability as major barriers to operational deployment. Nature[University of Liverpool]livrepository.liverpool.ac.ukUniversity of LiverpoolExplainable artificial intelligence for mental health through…by DW Joyce · 2023 · Cited by 265 — Across health…

Accuracy Alone Is Not Enough

A common argument in favour of black-box systems is straightforward: if they consistently outperform human experts or simpler models, why demand complete explanations?

In some situations, this argument has force. Forecasting is ultimately judged by outcomes. If a model repeatedly predicts hospital admissions, influenza spread, or dengue outbreaks more accurately than conventional methods, agencies gain a practical reason to use it. During COVID-19, forecasting hubs assembled predictions from many modelling teams, and ensemble approaches often performed better than individual forecasts. Recent AI systems have continued to demonstrate improvements in short-term epidemic prediction.[arXiv]arxiv.orgarXiv The Three Ghosts of Medical AI: Can the Black-Box Present Deliver?arXiv The Three Ghosts of Medical AI: Can the Black-Box Present Deliver? [3Nature 3PMC]

However, public-health decisions differ from many commercial prediction tasks.

A retailer can quietly adjust inventory if an algorithm makes a mistake. A health ministry may impose travel restrictions, redirect vaccines, close schools, or issue emergency warnings affecting millions of people. The threshold for trust is therefore higher.

Officials often need more than a prediction. They need confidence that the forecast remains reliable when conditions change. They need to know whether uncertainty is growing. They need to identify which assumptions matter most. Pure predictive performance on historical benchmarks may not answer these questions.[World Health Organization]WikipediaWorld Health OrganizationThe World Health Organization (WHO) is a specialized agency of the United Nations (UN) which coordinates resp…[Iris]iris.who.intIris Ethics and governance of artificial intelligence for healthWhether AI can advance the interests of patients and communities depends on a collective effort to design and implement ethically def…

This is especially important because outbreak forecasting frequently operates under conditions that differ from the past. The most valuable warning is often the one about an emerging threat that has never been seen before.

The COVID-19 Lesson: Many Models, No Single Oracle

One of the most important governance lessons from COVID-19 is that agencies rarely relied on a single forecasting system.

The US COVID-19 Forecast Hub collected forecasts from dozens of modelling teams using different methods, assumptions, and data sources. Rather than treating any one model as authoritative, forecasters increasingly combined predictions into ensemble forecasts designed to reduce individual model errors.[COVID-19 Forecast Hub]covid19forecasthub.orgCOVID-19 Forecast HubCOVID 19 forecast hub: HomeFrom 2020 to 2024, this site collected real-time forecasts of COVID-19 hospitalizations… Nature This approach reflects a deeper institutional reality: public-health agencies generally trust forecasting systems more when they can compare[nature.com]nature.comPersonalized health monitoring using explainable AIby MS Vani · 2025 · Cited by 65 — However, worried about the trust, accountabili… them against competing models.

A black-box model that produces strong forecasts can still be useful if:

In practice, agencies often place more trust in a system that has repeatedly demonstrated reliable performance than in one that merely offers elegant explanations.

Yet the COVID period also showed how rapidly forecasting accuracy can degrade when behaviour changes, new variants emerge, testing patterns shift, or policy interventions alter transmission dynamics. Models that performed well during one phase of a pandemic sometimes struggled during another.[PMC]pmc.ncbi.nlm.nih.govthe black box: A systematic review of Explainable…by D Muhammad · 2024 · Cited by 246 — This systematic literature review examines sta…[Nature]nature.comPersonalized health monitoring using explainable AIby MS Vani · 2025 · Cited by 65 — However, worried about the trust, accountabili…

The lesson is not that forecasting failed. It is that outbreak prediction remains inherently uncertain, and sophisticated AI does not eliminate that uncertainty.

Trust and Accountability in Outbreak Response

The strongest objections to black-box forecasting are usually governance concerns rather than technical ones.

If an agency acts on an AI warning that later proves wrong, who is responsible?

Possible answers include:

This ambiguity becomes dangerous when decisions carry major social consequences. Trust in public-health institutions depends partly on their ability to explain why actions were taken. An unexplained algorithmic recommendation can weaken that legitimacy.[World Health Organization]WikipediaWorld Health OrganizationThe World Health Organization (WHO) is a specialized agency of the United Nations (UN) which coordinates resp…[Iris]iris.who.intIris Ethics and governance of artificial intelligence for healthWhether AI can advance the interests of patients and communities depends on a collective effort to design and implement ethically def…

The World Health Organization has repeatedly emphasised that AI systems in health should support, rather than replace, human decision-making. Its governance guidance stresses human autonomy, accountability, transparency, safety, and public interest as core principles for AI deployment.[World Health Organization]WikipediaWorld Health OrganizationThe World Health Organization (WHO) is a specialized agency of the United Nations (UN) which coordinates resp…[World Health Organization]WikipediaWorld Health OrganizationThe World Health Organization (WHO) is a specialized agency of the United Nations (UN) which coordinates resp…

For outbreak response, this suggests a practical rule: officials should remain accountable for decisions even when AI contributes to them.

That principle may sound obvious, but it has important consequences. Agencies must maintain enough internal expertise to challenge forecasts rather than simply accepting them. Otherwise responsibility becomes symbolic rather than real.

Explainable AI Helps, But It Does Not Fully Solve the Problem

Researchers have invested heavily in explainable AI, often called XAI. These methods attempt to reveal which variables influenced a prediction or identify patterns driving a model’s conclusions. Techniques such as SHAP values, feature attribution methods, and attention visualisations are increasingly used in healthcare AI.[PMC]pmc.ncbi.nlm.nih.govPMCAccuracy of US CDC COVID-19 forecasting modelsby A Chharia · 2024 · Cited by 21 — In this study, we systematically analyze all US CDC COVID-19 forecasting models, by first categori…[PMC]pmc.ncbi.nlm.nih.govby G Chassang · 2025 · Cited by 8 — This paper discusses the responsible use of artificial intelligence (AI) in public health and in m…

Explainability can improve trust in several ways:

However, explainability has limits.

A system may produce convincing explanations that are only partial descriptions of what the model is actually doing. Some researchers argue that post-hoc explanations can create an illusion of understanding rather than genuine transparency. Even highly interpretable visualisations do not necessarily guarantee that a model will behave reliably under new conditions.[University of Liverpool]livrepository.liverpool.ac.ukUniversity of LiverpoolExplainable artificial intelligence for mental health through…by DW Joyce · 2023 · Cited by 265 — Across health…[PMC]pmc.ncbi.nlm.nih.govUnbox the black-box for the medical explainable AI via multi…by G Yang · 2022 · Cited by 887 — Explainable Artificial Intelligence…

For that reason, many public-health experts increasingly treat explainability as one component of trustworthiness rather than a complete solution.

When Black-Box Models Are Most Useful

The strongest case for using opaque forecasting systems appears in relatively narrow situations.

These include:

In these settings, agencies can compare forecasts against known outcomes and continuously measure performance. The consequences of individual prediction errors may also be easier to manage.[Public Health AI Handbook]publichealthaihandbook.comPublic Health AI Handbook Epidemic Forecasting with AICOVID-19 experiments, PandemicLLM reported better 1- to 3-week forecasting performance than several CDC COVID-19 Forecast Hub baselines… Nature The weakest case for reliance emerges when:[nature.com]nature.comPersonalized health monitoring using explainable AIby MS Vani · 2025 · Cited by 65 — However, worried about the trust, accountabili…

• A pathogen is genuinely novel.
• Data quality is poor.
• Forecasts drive highly disruptive interventions.
• Independent validation is unavailable.
• Decision-makers do not understand the model’s limitations.

The same model that provides useful guidance for seasonal influenza may be far less trustworthy during the first weeks of an unfamiliar pandemic.

Balancing Automation With Epidemiological Judgement

The most promising governance model is neither full automation nor full rejection.

Instead, many public-health institutions are moving toward a “human-plus-machine” approach. Forecasts become decision-support tools rather than decision-makers.

In practice, this means AI systems can generate hypotheses, risk estimates, and early warnings, while epidemiologists evaluate whether the outputs fit biological realities, surveillance evidence, and local conditions. The forecast becomes one input among several rather than an unquestioned instruction.[CDC]cdc.govCenter for Forecasting and Outbreak Analytics | CFACFA uses advanced analytic approaches, like forecasting and modeling, to drive effe…[Restored CDC]restoredcdc.orgCFA: Behind the ModelOct 4, 2024 — The models helped forecast the expected size and duration of the outbreak and helped the Chicago Depar…

This approach also creates institutional resilience. Human experts can detect situations where models appear to be drifting. Models can process more information than human teams can analyse manually. Each compensates for weaknesses in the other.

Importantly, this arrangement aligns with a broader vision of AI-enhanced human flourishing. The most valuable role for advanced forecasting systems may not be replacing public-health judgement but expanding it. If future AI systems help detect outbreaks earlier, integrate global surveillance data, simulate intervention strategies, and reduce uncertainty during crises, they could strengthen civilisation’s capacity to prevent catastrophe. But that benefit depends on institutions remaining capable of understanding, auditing, and governing the systems they use.

What Public Health Agencies Should Actually Demand

The key question is not whether a model is technically a black box. The more important question is whether it is trustworthy enough for its intended use.

Before relying on an outbreak forecasting system, agencies should generally expect:

• Independent validation against real-world outcomes.
• Transparent reporting of uncertainty.
• Regular performance monitoring.
• Documentation of training data and limitations.
• Human oversight by epidemiologists and public-health officials.
• Comparison against alternative models rather than reliance on a single forecast source.
• Clear procedures for responding when forecasts fail.[ICTworks]ictworks.orgwho guidance artificial intelligence healthProtecting human autonomy · 2. Promoting human well-being, safety, and public interest. · 3. Ensuring…Read more…[World Health Organization]WikipediaWorld Health OrganizationThe World Health Organization (WHO) is a specialized agency of the United Nations (UN) which coordinates resp…[World Health Organization]WikipediaWorld Health OrganizationThe World Health Organization (WHO) is a specialized agency of the United Nations (UN) which coordinates resp…

Under those conditions, even partially opaque systems can become useful public-health tools.

Without those safeguards, black-box forecasting risks creating a dangerous situation in which agencies gain powerful predictions but lose the ability to judge when those predictions deserve confidence. The future of pandemic forecasting is therefore likely to depend less on finding a perfect predictive model than on building institutions capable of combining advanced AI with transparency, accountability, and human expertise.[Iris]iris.who.intIris Ethics and governance of artificial intelligence for healthWhether AI can advance the interests of patients and communities depends on a collective effort to design and implement ethically def…[PMC]pmc.ncbi.nlm.nih.govthe black box: A systematic review of Explainable…by D Muhammad · 2024 · Cited by 246 — This systematic literature review examines sta…

Amazon book picks

Further Reading

Books and field guides related to Can Officials Trust AI They Cannot Fully Explain?. Use these as the next step if you want deeper reading beyond the article.

Book

The Alignment Problem

By Brian Christian

Addresses explainability, objectives and trust in AI systems.

See on Amazon

Book

Hello World

By Hannah Fry

Explores algorithmic decision-making, trust and transparency.

See on Amazon

Book

The Signal and the Noise

By Nate Silver

Includes discussion of forecasting uncertainty and public decision-making.

See on Amazon

Book

Superforecasting

By Philip E. Tetlock, Dan Gardner

Useful for interpreting uncertain outbreak forecasts.

See on Amazon

Browse more on Amazon:The Alignment ProblemHello WorldThe Signal and the Noise

As an Amazon Associate I earn from qualifying purchases.

eBay marketplace picks

Marketplace Samples

Example marketplace items related to this page. Use the search link to explore similar finds on eBay.

Shop location

UsingUSA USA USAUKCanadaAustraliaIreland

Example eBay listing

Resident Evil T-Virus LED Lamp Night light Glow in the Dark Collectible Model

SearcheBay.co.uk: virus model

Browse similar oneBay.co.uk

Example eBay listing

MVP Virus Disc Golf Disc Pink 172g Distance Driver Virus Model PDGA Approved

SearcheBay.co.uk: virus model

Browse similar oneBay.co.uk

Example eBay listing

Virus Model Framed Wall Art Poster Canvas Print Picture

SearcheBay.co.uk: virus model

Browse similar oneBay.co.uk

Example eBay listing

Resident Evil Nemisis - Resin Model / Figure / Statue Fan Art T-Virus S.T.A.R.S

SearcheBay.co.uk: virus model

Browse similar oneBay.co.uk

Browse more oneBay.co.uk

Example items shown for inspiration; availability and pricing can change. Branchoria may earn a commission if you purchase through outbound eBay links.

Endnotes

1. Source: cdc.gov
Link:https://www.cdc.gov/forecast-outbreak-analytics/index.html

2. Source: sciencedirect.com
Link:https://www.sciencedirect.com/science/article/pii/S2667102623000578

3. Source: pmc.ncbi.nlm.nih.gov
Link:https://pmc.ncbi.nlm.nih.gov/articles/PMC8459787/

4. Source: arxiv.org
Title: arXiv The Three Ghosts of Medical AI: Can the Black-Box Present Deliver?
Link:https://arxiv.org/abs/2012.06000

5. Source: nature.com
Link:https://www.nature.com/articles/s41598-025-15867-z

6. Source: pmc.ncbi.nlm.nih.gov
Link:https://pmc.ncbi.nlm.nih.gov/articles/PMC11382209/

7. Source: nature.com
Link:https://www.nature.com/articles/s41597-022-01517-w

8. Source: pmc.ncbi.nlm.nih.gov
Title: PMCAccuracy of US CDC COVID-19 forecasting models
Link:https://pmc.ncbi.nlm.nih.gov/articles/PMC11233691/

9. Source: arxiv.org
Title: arXiv Improving Outbreak Forecasts Through Model Augmentation
Link:https://arxiv.org/abs/2506.16410

10. Source: who.int
Link:https://www.who.int/publications/i/item/9789240029200

11. Source: iris.who.int
Title: Iris Ethics and governance of artificial intelligence for health
Link:https://iris.who.int/server/api/core/bitstreams/4f2c477c-4b72-4ca1-9a78-a1e73af64e50/content

12. Source: covid19forecasthub.org
Link:https://covid19forecasthub.org/

13. Source: who.int
Link:https://www.who.int/publications/i/item/9789240037403

14. Source: ictworks.org
Title: who guidance artificial intelligence health
Link:https://www.ictworks.org/who-guidance-artificial-intelligence-health/

15. Source: arxiv.org
Title: arXiv From Black Box to Insight: Explainable AI for Extreme Event Preparedness
Link:https://arxiv.org/abs/2511.13712

16. Source: pmc.ncbi.nlm.nih.gov
Link:https://pmc.ncbi.nlm.nih.gov/articles/PMC12027014/

17. Source: who.int
Link:https://www.who.int/

18. Source: who.int
Link:https://www.who.int/publications/i/item/9789240084759

19. Source: arxiv.org
Link:https://arxiv.org/html/2508.12260v5

20. Source: publichealthaihandbook.com
Title: Public Health AI Handbook Epidemic Forecasting with AI
Link:https://publichealthaihandbook.com/applications/forecasting.html

21. Source: phgfoundation.org
Link:https://www.phgfoundation.org/wp-content/uploads/2023/10/black-box-machine-learning-landscape.pdf

22. Source: livrepository.liverpool.ac.uk
Link:https://livrepository.liverpool.ac.uk/3170961/1/Explainable%20artificial%20intelligence%20for%20mental%20health%20through%20transparency%20and%20interpretability%20for%20understandability.pdf

23. Source: github.com
Link:https://github.com/CDCgov/covid19-forecast-hub

24. Source: restoredcdc.org
Link:https://restoredcdc.org/www.cdc.gov/cfa-behind-the-model/php/behind-the-model/index.html

25. Source: Wikipedia
Title: World Health Organization
Link:https://en.wikipedia.org/wiki/World_Health_Organization

26. Source: aiforgood.itu.int
Title: int World Health Organization (WHO)
Link:https://aiforgood.itu.int/about-us/un-ai-actions/who/

Additional References

27. Source: researchgate.net
Link:https://www.researchgate.net/publication/340495230_Explainable_AI_for_Healthcare_From_Black_Box_to_Interpretable_Models

28. Source: reliefweb.int
Link:https://reliefweb.int/report/world/artificial-intelligence-health-emergencies-who-advances-public-health-intelligence-and-surveillance-through-innovation

29. Source: eurasiareview.com
Title: 30062021 who issues first global report in artificial intelligence in health
Link:https://www.eurasiareview.com/30062021-who-issues-first-global-report-in-artificial-intelligence-in-health/

30. Source: privacyinternational.org
Title: our analysis who report ethics and governance artificial intelligence health
Link:https://privacyinternational.org/news-analysis/4594/our-analysis-who-report-ethics-and-governance-artificial-intelligence-health

31. Source: researchprofiles.ku.dk
Title: ethics and governance of artificial intelligence for health who g
Link:https://researchprofiles.ku.dk/en/publications/ethics-and-governance-of-artificial-intelligence-for-health-who-g/

32. Source: hub.jhu.edu
Title: artificial intelligence infectious disease forecasting
Link:https://hub.jhu.edu/2025/06/06/artificial-intelligence-infectious-disease-forecasting/

33. Source: youtube.com
Title: Enhancing Long-Term Forecasting: Learning from COVID-19 Models
Link:https://www.youtube.com/watch?v=5-nVT5sWJI4

34. Source: ncdirindia.org
Link:https://www.ncdirindia.org/Downloads/WHO_AI_Ethics.pdf

35. Source: youtube.com
Title: From the Unseen to Foresight: Epidemic Modeling in the Age of Computing and AI
Link:https://www.youtube.com/watch?v=gD0obKGIWVk

36. Source: youtube.com
Title: Presentation on Enhancing Long-term Forecasting: Learning from COVID-19 Models
Link:https://www.youtube.com/watch?v=7c8K7FQwD4Q