Trust Limits

Introduction

One of the most hopeful arguments for advanced AI is that it could make expert knowledge cheap and widely available. A farmer with a smartphone might gain agronomy advice once reserved for specialists. A patient in a remote area could receive medical guidance instantly. A small business owner might navigate regulations without paying a solicitor or consultant.

Trust Limits illustration 1 But there is a sharp difference between access to answers and access to dependable expertise.

Large language models are designed to produce plausible, fluent responses. They are often useful, sometimes remarkably so. Yet they can also invent facts, misread situations, reproduce hidden biases, or give confident advice without understanding the real-world consequences. In low-stakes situations this may be annoying. In medicine, law, finance, welfare systems, or public administration, it can become dangerous.

This creates one of the central tensions in the broader vision of AI-enabled abundance. If AI is to help spread knowledge at civilisation scale, societies must solve a harder problem than making answers cheap. They must build systems that make cheap advice trustworthy, accountable, and safe enough to rely on.

Why fluent AI advice can be misleading

Traditional software usually fails visibly. A spreadsheet crashes. A database returns an error. Large language models fail differently. They often produce incorrect answers that sound authoritative.

Researchers call these failures “hallucinations”: outputs that are fabricated, inaccurate, or unsupported. But the problem is broader than random mistakes. These systems generate language by predicting statistically plausible text, not by verifying truth in the way a human expert ideally would.

That matters because people naturally interpret conversational fluency as competence.

A chatbot that speaks like a doctor or lawyer encourages users to assume it understands medicine or law in a deep sense. In reality, it may simply be producing convincing patterns from training data. Even advanced systems can struggle with:

unusual or ambiguous cases
missing local information
recent legal or medical changes
conflicting evidence
regional regulations
hidden assumptions in the user’s question
gaps in training data
cultural or linguistic context

This creates a dangerous mismatch between appearance and reliability. A weak internet search at least looks uncertain. A conversational AI system often sounds certain even when it is wrong.

The risk becomes greater precisely because the advice is cheap and immediate. Historically, expert scarcity acted as a brake on misuse. People hesitated before replacing a doctor or solicitor with guesswork. AI systems reduce that friction dramatically.

Hallucinations are not rare edge cases

Early public debates sometimes treated hallucinations as temporary bugs that would quickly disappear. Evidence from legal, medical, and administrative use suggests the issue is more structural.

A widely discussed 2023 US legal case involved lawyers submitting fictitious court citations generated by ChatGPT. The court later sanctioned the lawyers after discovering that the cited cases did not exist. [The Guardian]theguardian.comtwo us lawyers fined submitting fake court citations chatgptThe GuardianTwo US lawyers fined for submitting fake court citations…23 Jun 2023 — A US judge has fined two lawyers and a law firm $5…

The problem has continued well beyond the first wave of publicity. Reuters reported multiple 2025 and 2026 incidents involving fabricated citations, phantom quotations, and incorrect legal arguments generated with AI systems and submitted to courts by practising lawyers. [Reuters]reuters.comMichael Morgan, a lawyer from Morgan & Morgan, from participating in litigation against Harvard Medical School over a scandal involving s… [Reuters]reuters.comfederal judge for submitting a court filing that included fictitious quotations generated by artificial intelligence. The AI tool, Anthro… [Reuters]reuters.comBankruptcy Court Judge Martin Glenn for submitting a court filing containing AI-generated inaccuracies. The errors, referred to as "hallu…

Researchers studying legal AI systems have found that hallucinations remain common even in specialised legal products marketed as safer than general chatbots. One Stanford-led study found substantial hallucination rates across major legal AI research tools despite vendor claims about reliability improvements. [arXiv]arxiv.orgarXivHallucination-Free? Assessing the Reliability of Leading AI Legal Research ToolsMay 30, 2024…Published: May 30, 2024

This matters beyond the legal profession because law provides unusually clear evidence of the underlying mechanism. Court citations can be checked directly. In many other domains, especially medicine or bureaucracy, incorrect advice may go unnoticed.

A patient may never learn that an AI system missed an uncommon diagnosis. A welfare claimant may not discover that a chatbot misunderstood local eligibility rules. A small business owner may follow incorrect tax guidance for months before facing penalties.

The danger is not merely isolated mistakes. It is the possibility of silent failure at enormous scale.

Missing local context is often the real problem

Many AI failures do not come from pure fabrication. They come from missing situational understanding.

Human experts rely heavily on tacit context:

local laws
institutional procedures
cultural expectations
practical constraints
non-verbal signals
incomplete or contradictory evidence
awareness of what is uncertain

AI systems often lack this grounding.

A medical model trained heavily on data from wealthy countries may perform poorly in regions with different disease prevalence, infrastructure limitations, or treatment availability. A legal chatbot may explain federal rules while ignoring local regulations that actually determine the outcome. An AI tutor may give technically correct explanations that fail to match the student’s educational background or language level.

These problems become especially serious when users cannot easily detect the mistake themselves.

A trained cardiologist might spot an AI error quickly. A frightened patient probably cannot.

This creates a paradox in the dream of democratised expertise. The people who benefit most from cheap AI advice are often the least equipped to verify it independently.

Bias can scale faster than expertise

Cheap AI advice can also spread hidden biases more efficiently than older systems.

Training data reflects unequal societies. Historical medical data may underrepresent women, minority groups, or poorer populations. Legal data may reflect unequal policing or sentencing practices. Educational datasets may privilege dominant languages and cultural assumptions.

As a result, AI systems can inherit distortions from the world they learned from.

The World Health Organization has repeatedly warned that AI systems in healthcare can amplify bias, deepen inequality, and create new risks if deployed without strong governance and oversight. [World Health Organization]WikipediaWorld Health OrganizationThe World Health Organization (WHO) is a specialized agency of the United Nations (UN) which coordinates resp… [World Health Organization]WikipediaWorld Health OrganizationThe World Health Organization (WHO) is a specialized agency of the United Nations (UN) which coordinates resp…

Bias problems are especially dangerous when combined with the authority effect of AI systems. Users often assume machine-generated recommendations are objective or neutral. In practice, automated systems can hide subjective assumptions behind a veneer of technical legitimacy.

The scale issue matters here too. A biased human expert may affect hundreds of people. A biased AI system integrated into public services could affect millions.

Medicine shows the highest stakes

Healthcare is one of the strongest cases for AI-enabled abundance. In principle, AI systems could extend medical expertise to regions with severe doctor shortages, accelerate diagnosis, reduce costs, and improve preventive care.

But medicine also shows why reliability and accountability matter so much.

Medical decisions are highly context-sensitive. Symptoms that look minor in one patient may signal an emergency in another. Drug interactions, medical history, age, pregnancy, disability, and coexisting conditions all matter.

A language model may provide generally reasonable advice while missing exactly the detail that changes the outcome.

This is why most medical regulators and health organisations treat current generative AI systems as support tools rather than autonomous clinicians. The World Health Organization has emphasised the need for transparency, human oversight, validation, and clear accountability before large-scale deployment in healthcare. [World Health Organization]WikipediaWorld Health OrganizationThe World Health Organization (WHO) is a specialized agency of the United Nations (UN) which coordinates resp…

There is also a deeper institutional issue. Modern medicine is not simply the delivery of information. It includes:

professional licensing
malpractice liability
audit trails
peer review
clinical standards
informed consent
systems for correcting mistakes

An AI chatbot alone does not provide those safeguards.

The optimistic vision of abundant medical expertise therefore depends less on raw model capability than on whether societies can build trustworthy supervisory systems around the models.

Trust Limits illustration 2

Bureaucracy becomes risky when nobody is responsible

AI advice failures are especially difficult in areas where accountability is already diffuse.

A doctor can lose a licence. A solicitor can face sanctions. A civil servant answers to institutional procedures. But conversational AI systems complicate responsibility.

If a chatbot gives harmful advice:

Is the developer responsible?
The organisation deploying it?
The human operator?
The user?
Nobody?

Real-world disputes are already emerging around this question.

In 2024, a Canadian tribunal ruled against Air Canada after the airline’s chatbot gave a passenger incorrect information about bereavement fare policies. The tribunal rejected the company’s attempt to treat the chatbot as separately responsible and held the airline accountable for the misinformation. [Ars Technica]arstechnica.comair canada must honor refund policy invented by airlines chatbotAir Canada must honor refund policy invented by airline's…16 Feb 2024 — Air Canada was forced to give a partial refund to a grieving p… [3The Guardian 3Forbes]

The case mattered because it exposed a larger governance problem. Organisations may gain large efficiency benefits from AI systems while attempting to shift responsibility for errors onto users or the technology itself.

That approach becomes dangerous if AI advice increasingly mediates access to welfare, healthcare, insurance, immigration systems, legal rights, or financial services.

A society cannot safely replace human expertise with automated guidance unless responsibility remains legible.

The hidden risk of overreliance

One of the most important dangers is not that AI systems always fail. It is that they work often enough to encourage overconfidence.

Pilots using autopilot systems sometimes become less attentive during rare emergencies. Similar patterns can emerge with AI assistants.

Researchers studying legal AI warn about “automation bias”: the tendency for humans to trust machine-generated outputs too readily, especially when the system appears sophisticated or authoritative. [arXiv]arxiv.orgarXivHallucination-Free? Assessing the Reliability of Leading AI Legal Research ToolsMay 30, 2024…Published: May 30, 2024

This creates a subtle degradation problem:

professionals may stop checking carefully
junior workers may lose opportunities to develop expertise
institutions may become dependent on systems they do not fully understand
users may defer to AI even when human judgement is needed

Over time, societies could end up with more access to answers but less deeply rooted human expertise.

That would undermine one of the central promises of AI-enabled flourishing. Civilisations become more resilient when knowledge is distributed, understood, and corrigible. They become more fragile when crucial systems depend on opaque tools few people can properly evaluate.

Trust Limits illustration 3

Cheap advice only becomes expertise through institutions

The key lesson from current failures is that AI-generated answers are not equivalent to trustworthy expertise on their own.

Reliable expertise is institutional as much as technical.

Modern professions evolved around mechanisms that reduce error:

licensing
supervision
peer review
professional norms
insurance

audit systems
second opinions
appeals processes
liability rules
evidence standards

The most plausible path toward genuinely abundant expertise may therefore involve hybrid systems rather than fully autonomous advisers.

Examples include:

AI-assisted clinicians who remain legally responsible for decisions
legal drafting systems tied directly to verified case databases
AI tutors monitored by human teachers
government chatbots with clear escalation routes to human staff
medical systems that expose uncertainty levels rather than hiding them
audit trails showing how recommendations were generated
region-specific models trained on local laws and practices

The goal is not simply to make advice faster. It is to make dependable judgement scalable without removing human accountability.

The deeper challenge for an AI-enabled civilisation

The broader “AI bloom” vision depends heavily on the idea that intelligence and expertise can become abundant rather than scarce.

If successful, that could transform education, healthcare, science, governance, and economic opportunity. Entire populations currently excluded from high-quality expertise could gain access to capabilities once available only to wealthy institutions or rich countries.

But the danger of cheap AI advice reveals an important limit to technological optimism.

Human flourishing does not come merely from generating more information. It depends on trust, verification, institutions, incentives, and the ability to correct mistakes before they scale catastrophically.

A world flooded with persuasive but unreliable advice could become less governable rather than more enlightened. The same systems that democratise expertise could also democratise confusion, manipulation, bureaucratic error, and institutional evasion of responsibility.

The optimistic future therefore requires more than smarter models. It requires social systems capable of distinguishing between fluent output and dependable knowledge.

That may become one of the defining governance challenges of an AI-rich century.

Endnotes

Source: forbes.com
Title: Lawyer Used Chat GPT In Court—And Cited Fake Cases
Link: https://www.forbes.com/sites/mollybohannon/2023/06/08/lawyer-used-chatgpt-in-court-and-cited-fake-cases-a-judge-is-considering-sanctions/
Source snippet
A...8 Jun 2023 —... cases of AI “hallucinations” making it to court. ChatGPT Phone App. A lawyer who used ChatGPT in court and ended up...
Source: reuters.com
Link: https://www.reuters.com/legal/litigation/lawyer-barred-harvard-morgue-scandal-case-over-fake-ai-citations-2026-05-19/
Source snippet
Michael Morgan, a lawyer from Morgan & Morgan, from participating in litigation against Harvard Medical School over a scandal involving s...
Source: reuters.com
Link: https://www.reuters.com/legal/transactional/lawyer-apologizes-phantom-ai-quotes-trump-layoffs-case-2026-05-18/
Source snippet
federal judge for submitting a court filing that included fictitious quotations generated by artificial intelligence. The AI tool, Anthro...
Source: reuters.com
Link: https://www.reuters.com/legal/litigation/sullivan-cromwell-law-firm-apologizes-ai-hallucinations-court-filing-2026-04-21/
Source snippet
Bankruptcy Court Judge Martin Glenn for submitting a court filing containing AI-generated inaccuracies. The errors, referred to as "hallu...
Source: reuters.com
Title: US appeals court fines lawyers $30,000 in latest AI-related sanction A U.S
Link: https://www.reuters.com/legal/litigation/us-appeals-court-fines-lawyers-30000-latest-ai-related-sanction-2026-03-16/
Source snippet
federal appeals court has fined attorneys Van Irion and Russ Egli $30,000 for submitting legal filings containing over two dozen fake cas...
Source: arxiv.org
Link: https://arxiv.org/abs/2405.20362
Source snippet
arXivHallucination-Free? Assessing the Reliability of Leading AI Legal Research ToolsMay 30, 2024...

Published: May 30, 2024
Source: arxiv.org
Link: https://arxiv.org/abs/2401.01301
Source: who.int
Link: https://www.who.int/publications/i/item/9789240084759
Source snippet
World Health OrganizationEthics and governance of artificial intelligence for health25 Mar 2025 — This guidance addresses one type of gen...
Source: who.int
Link: https://www.who.int/publications/i/item/9789240029200
Source snippet
World Health OrganizationEthics and governance of artificial intelligence for health28 Jun 2021 — The report identifies the ethical chall...
Source: forbes.com
Title: what air canada lost in remarkable lying ai chatbot case
Link: https://www.forbes.com/sites/marisagarcia/2024/02/19/what-air-canada-lost-in-remarkable-lying-ai-chatbot-case/
Source snippet
Following the...Read more...
Source: arxiv.org
Link: https://arxiv.org/abs/2603.15937
Source snippet
arXivWhy Avoid Generative Legal AI Systems? Hallucination, Overreliance, and their Impact on ExplainabilityMarch 16, 2026...

Published: March 16, 2026
Source: arxiv.org
Link: https://arxiv.org/abs/2603.23857
Source: who.int
Link: https://www.who.int/
Source: reuters.com
Link: https://www.reuters.com/technology/artificial-intelligence/ai-hallucinations-court-papers-spell-trouble-lawyers-2025-02-18/
Source snippet
AI 'hallucinations' in court papers spell trouble for lawyers18 Feb 2025 — AI sometimes produces false information, known as "hallucinations...
Source: reuters.com
Title: new york lawyers sanctioned using fake chatgpt cases legal brief 2023 06 22
Link: https://www.reuters.com/legal/new-york-lawyers-sanctioned-using-fake-chatgpt-cases-legal-brief-2023-06-22/
Source snippet
New York lawyers sanctioned for using fake ChatGPT...26 Jun 2023 — The judge found the lawyers acted in bad faith and made "acts of cons...
Source: hai.stanford.edu
Title: ai trial legal models hallucinate 1 out 6 or more benchmarking queries
Link: https://hai.stanford.edu/news/ai-trial-legal-models-hallucinate-1-out-6-or-more-benchmarking-queries
Source snippet
hallucinate,” or make up false information. In one highly-publicized case, a New York lawyer faced sanctions for citing ChatGPT-invented...
Source: theguardian.com
Title: two us lawyers fined submitting fake court citations chatgpt
Link: https://www.theguardian.com/technology/2023/jun/23/two-us-lawyers-fined-submitting-fake-court-citations-chatgpt
Source snippet
The GuardianTwo US lawyers fined for submitting fake court citations...23 Jun 2023 — A US judge has fined two lawyers and a law firm $5...
Source: theguardian.com
Title: The Guardian Air Canada ordered to pay customer who was misled by
Link: https://www.theguardian.com/world/2024/feb/16/air-canada-chatbot-lawsuit
Source snippet
The GuardianAir Canada ordered to pay customer who was misled by...February 16, 2024 — 16 Feb 2024 — Canada's largest airline has been o...

Published: February 16, 2024
Source: arstechnica.com
Title: air canada must honor refund policy invented by airlines chatbot
Link: https://arstechnica.com/tech-policy/2024/02/air-canada-must-honor-refund-policy-invented-by-airlines-chatbot/
Source snippet
Air Canada must honor refund policy invented by airline's...16 Feb 2024 — Air Canada was forced to give a partial refund to a grieving p...
Source: Wikipedia
Title: World Health Organization
Link: https://en.wikipedia.org/wiki/World_Health_Organization
Source snippet
World Health OrganizationThe World Health Organization (WHO) is a specialized agency of the United Nations (UN) which coordinates resp...
Source: theguardian.com
Title: utah lawyer chatgpt ai court brief
Link: https://www.theguardian.com/us-news/2025/may/31/utah-lawyer-chatgpt-ai-court-brief
Source snippet
US lawyer sanctioned after being caught using ChatGPT...1 Jun 2025 — The Utah court of appeals has sanctioned a lawyer after he was disc...
Source: scirp.org
Link: https://www.scirp.org/reference/referencespapers?referenceid=3997612
Source snippet
World Health Organization (WHO) (2021) Ethics and...We conclude that successful AI integration in healthcare depends on adaptive regulat...

Additional References

Source: researchgate.net
Link: https://www.researchgate.net/publication/404303406_AI_Hallucinations_as_a_Form_of_Harm_An_Analysis_under_WHO_Guidance_on_Ethics_and_Governance_of_Artificial_Intelligence_for_Health
Source snippet
AI Hallucinations as a Form of Harm: An Analysis under...5 Apr 2026 — Drawing on WHO Principle 5.2, requiring protection from mental and...
Source: airbnb.co.uk
Link: https://www.airbnb.co.uk/
Source snippet
Airbnb | Holiday rentals, cabins, beach houses & moreGet an Airbnb for every kind of trip → 7 million holiday rentals → 2 million Guest F...
Source: concetticontrastivi.org
Link: https://concetticontrastivi.org/wp-content/uploads/2024/11/ethics-and-governance-of-artificial-intelligence-for-health.pdf
Source snippet
Ethics and governance of artificial intelligence for healthFalse responses, known colloquially as “hallucinations”, are indistinguishable...
Source: goldbergsegalla.com
Link: https://www.goldbergsegalla.com/app/uploads/2023/10/Fake-Cases-Real-Consequences-Misuse-of-ChatGPT-Christoper-F.-Lyon-NY-Litigator.pdf
Source snippet
Fake Cases, Real Consequences: Misuse of ChatGPT...by CF Lyon · Cited by 7 — Because the fake case opinions generated by ChatGPT contain...
Source: medium.com
Link: https://medium.com/the-generator/the-fate-of-the-lawyer-who-cited-fake-cases-from-chatgpt-772acc4f99ca
Source snippet
The Fate of the Lawyer Who Cited Fake Cases from ChatGPTThe Fate of the Lawyer Who Cited Fake Cases from ChatGPT. It says a lot about how...
Source: biodiritto.org
Link: https://www.biodiritto.org/AI-Legal-Atlas/AHEAD-OBSERVATORY/AHEAD-Legislative-and-regulatory-framework/WHO-Ethics-ad-Governance-of-Artificial-Intelligence-for-Health-WHO-Guidance
Source snippet
WHO – Ethics ad Governance of Artificial Intelligence for...It recommends the development of policies, principles, and best practices fo...
Source: linkedin.com
Link: https://www.linkedin.com/posts/fdegni_who-ethics-and-governance-of-ai-for-health-activity-7313365824344408065-_8_9
Source snippet
WHO: Ethics and Governance of AI for health | Fabrizio DegniWorld Health Organization has published in March 2025 the "Ethics and Governa...

Published: March 2025
Source: americanbar.org
Title: bc tribunal confirms companies remain liable information provided ai chatbot
Link: https://www.americanbar.org/groups/business_law/resources/business-law-today/2024-february/bc-tribunal-confirms-companies-remain-liable-information-provided-ai-chatbot/
Source snippet
American Bar AssociationBC Tribunal Confirms Companies Remain Liable for...29 Feb 2024 — The British Columbia Civil Resolution Tribunal...
Source: gunnercooke.com
Title: ai fake cases and the courts a cautionary tale for the legal profession
Link: https://gunnercooke.com/ai-fake-cases-and-the-courts-a-cautionary-tale-for-the-legal-profession/
Source snippet
AI, Fake Cases, and the Courts: A Cautionary Tale for...19 May 2025 — Even Michael Cohen, former lawyer to Donald Trump, inadvertently s...

Published: May 2025
Source: esquiresolutions.com
Title: federal court turns up the heat on attorneys using chatgpt for research
Link: https://www.esquiresolutions.com/federal-court-turns-up-the-heat-on-attorneys-using-chatgpt-for-research/
Source snippet
Federal Court Turns Up the Heat on Attorneys Using...13 Aug 2025 — Dunn, the court declared that monetary sanctions are proving ineffect...