Within Verification
Who Pays When Legal AI Gets It Wrong?
Legal AI can accelerate research and drafting, but fabricated citations and incorrect authorities leave lawyers responsible for checking every consequential
On this page
- What legal AI systems still get wrong
- Why retrieval reduces but does not remove hallucinations
- How professional liability keeps human review essential
Page outline Jump by section
Introduction
Generative AI can dramatically speed up legal research, document drafting and case preparation, making legal services more accessible and potentially allowing lawyers to spend more time on strategy and client advice. That matters for any optimistic vision of AI-enabled human flourishing: if routine legal work becomes cheaper without sacrificing quality, more people could obtain legal help, businesses could resolve disputes more efficiently, and institutions could operate at lower cost.
The difficulty is that legal work depends on accuracy rather than plausibility. A contract clause, statutory reference or court citation is either correct or it is not. Modern AI systems can produce convincing but fabricated authorities, misstate legal rules or invent quotations while sounding highly confident. Courts have responded by making one principle unmistakably clear: AI may assist lawyers, but it does not assume their professional duties. Human verification remains legally indispensable, making the legal profession one of the clearest examples of why verification becomes scarcer—not less important—as AI-generated output becomes abundant.[American Bar Association]americanbar.orgAmerican Bar Association AI Hallucinations Are Real—and How to Avoid ThemAmerican Bar Association AI Hallucinations Are Real—and How to Avoid Them
What Legal AI Systems Still Get Wrong
Legal AI has improved rapidly, especially when combined with high-quality legal databases, yet several categories of error remain particularly important.
The most visible problem is fabricated authorities. A model may confidently cite cases that never existed, combine details from several genuine decisions into a fictional one, or attribute quotations to judges who never wrote them. Because the formatting resembles authentic legal citations, these mistakes can escape casual review.[American Bar Association]americanbar.orgAmerican Bar Association AI Hallucinations Are Real—and How to Avoid ThemAmerican Bar Association AI Hallucinations Are Real—and How to Avoid Them
Other common failure modes include:
- Incorrect descriptions of genuine precedents.
- Invented statutory provisions or procedural rules.
- Out-of-date legal analysis after legislation or case law changes.
- Failure to recognise jurisdictional differences.
- Confident answers where the law is genuinely uncertain or divided.
- Omission of important exceptions that substantially change the legal outcome.[arXiv]arxiv.orgOpen source on arxiv.org.
These failures matter because legal reasoning is cumulative. One fabricated authority can undermine an entire argument, damage counsel’s credibility, increase litigation costs and potentially prejudice a client’s case.
Research evaluating specialist legal AI systems suggests that domain-specific retrieval substantially reduces hallucinations compared with general-purpose chatbots, but does not eliminate them. Even commercial systems designed for legal practice continue to produce fabricated or inaccurate outputs often enough that independent verification remains necessary.[arXiv]arxiv.orgOpen source on arxiv.org.
The Court Cases That Changed the Conversation
The defining example remains Mata v. Avianca (2023).[section1983.org]section1983.orgmata v avianca incMata v. Avianca, Inc. — Case Law Library | section1983.org…
Lawyers representing a passenger opposing dismissal of his claim relied on ChatGPT during legal research. Their filing cited multiple cases that simply did not exist. When opposing counsel challenged the citations, the lawyers initially continued relying on the AI-generated material instead of independently verifying the authorities. After investigation, the federal court imposed sanctions and emphasised that existing professional obligations apply regardless of whether AI generated the work.[American Bar Association]americanbar.orgAmerican Bar Association AI Hallucinations Are Real—and How to Avoid ThemAmerican Bar Association AI Hallucinations Are Real—and How to Avoid Them
Mata became influential because the court’s reasoning extended beyond one technological mistake. The decision stressed that lawyers have always been responsible for verifying legal authorities before presenting them to a court. AI changed neither the ethical duty nor the allocation of responsibility.
Subsequent cases have reinforced the same principle across multiple jurisdictions. Courts have imposed financial sanctions, removed lawyers from cases, referred matters for disciplinary review, revoked temporary admission rights and publicly criticised practitioners who submitted AI-generated false authorities without adequate checking. Recent appellate decisions have continued to emphasise that reliance on AI cannot excuse failures to verify legal citations.[americanbar.org]americanbar.orgAmerican Bar AssociationGenAI and the Ethical Lawyer: Same Standards, Different Dilemmas Part II—How Courts Are Enforcing the Duty of Candor…
The pattern is increasingly clear: the legal system treats AI hallucinations not as technological accidents but as failures of professional supervision.
Why Retrieval Reduces but Does Not Remove Hallucinations
Many newer legal AI products rely on retrieval-augmented generation (RAG). Rather than relying solely on a language model’s internal statistical patterns, these systems first retrieve material from legal databases before generating an answer.
This architecture addresses an important weakness. If the model grounds its response in identifiable statutes, judgments or commentary, fabricated citations become less common.
However, retrieval introduces its own limits.
First, retrieved material may be incomplete or irrelevant if the search query is poorly framed.
Second, the model may accurately retrieve sources but incorrectly summarise or interpret them.
Third, legal questions often require combining multiple authorities, procedural rules and factual distinctions. The retrieval stage cannot guarantee that the model reasons correctly across all of them.
Independent evaluations of leading commercial legal AI products found substantially lower hallucination rates than general-purpose systems but still observed fabricated or inaccurate responses often enough to make marketing claims such as “hallucination-free” difficult to support.[arXiv]arxiv.orgOpen source on arxiv.org.
In practice, retrieval changes the probability of error rather than eliminating it.
Why Professional Liability Stays With the Lawyer
One of the most important legal consequences of generative AI is that responsibility has not moved alongside automation.
Courts generally assess filings by asking whether counsel exercised appropriate professional judgement, not whether an AI system produced the first draft.
That reflects several longstanding legal duties.
Duty of competence. Lawyers must provide competent representation using reasonable professional skill.
Duty of candour. Lawyers must not knowingly or negligently present false authorities or misleading information to the court.
Duty of supervision. Partners and supervising lawyers remain responsible for work produced under their authority, including AI-assisted drafting where appropriate.
Duty to the client. Lawyers owe clients competent advice regardless of which research tools they choose to use.[American Bar Association]americanbar.orgAmerican Bar AssociationGenAI and the Ethical Lawyer: Same Standards, Different Dilemmas Part II—How Courts Are Enforcing the Duty of Candor…
Consequently, professional liability does not disappear because AI generated the mistake. Depending on the circumstances, consequences may include:
- Court sanctions.
- Adverse costs orders.
- Professional disciplinary proceedings.
- Malpractice claims where client loss can be established.
- Damage to professional reputation and judicial credibility.[americanbar.org]americanbar.orgAmerican Bar AssociationGenAI and the Ethical Lawyer: Same Standards, Different Dilemmas Part II—How Courts Are Enforcing the Duty of Candor…
The underlying legal principle is familiar rather than novel: delegating work never delegates responsibility.
Why Verification Becomes the Scarce Resource
Legal practice illustrates the broader verification bottleneck particularly clearly.
A lawyer can now generate multiple contract drafts, research memoranda or litigation outlines within minutes. Producing alternatives has become inexpensive.
Checking them has not.
Meaningful verification often requires:
- Reading every cited authority.
- Confirming quotations against original judgments.
- Ensuring statutes remain current.
- Checking jurisdiction-specific rules.
- Testing whether unusual factual distinctions alter the analysis.
- Exercising professional judgement about litigation strategy.
These tasks require legal expertise rather than text generation. As AI accelerates drafting, proportionally more professional time shifts towards review, validation and responsibility.
Paradoxically, successful AI adoption may therefore increase the value of trusted reviewers. The bottleneck moves from writing documents to certifying that they are accurate enough to rely upon in proceedings where clients’ rights, money or liberty may be at stake.
What This Means for an AI-Enabled Future
Legal AI remains genuinely valuable despite these limitations.
It already helps lawyers summarise large document collections, identify potentially relevant authorities, draft routine correspondence, compare contract versions and generate first drafts more quickly than traditional workflows. Used carefully, these capabilities may reduce legal costs and improve access to justice by allowing professionals to devote more time to judgement-intensive work.
But the experience of legal hallucinations also offers a broader lesson for the idea of AI-driven abundance.
The optimistic case for AI is not simply that machines will generate unlimited information. It is that humans will be able to trust and act upon that information safely. In law, trust depends on accountability. Courts have consistently reinforced that AI is a tool for professional assistance, not a substitute for professional responsibility.
That distinction matters far beyond the legal profession. If advanced AI contributes to long-term human flourishing, it is likely to do so not by eliminating verification but by improving human capacity while preserving clear responsibility for consequential decisions. The legal system’s response suggests that, even in an age of abundant machine-generated knowledge, careful human judgement remains one of the scarcest and most valuable resources.
Amazon book picks
Further Reading
Books and field guides related to Who Pays When Legal AI Gets It Wrong?. Use these as the next step if you want deeper reading beyond the article.
The Alignment Problem: Machine Learning and Human Values
Finalist for the Los Angeles Times Book Prize A jaw-dropping exploration of everything that goes wrong when we build AI systems and the m...
Artificial Intelligence and Legal Analytics: New Tools for La...
First published 2017. Subjects: Artificial intelligence, Computers, law and legislation, Law, Methodology, Technological innovations.
The Future of the Professions: How Technology Will Transform...
First published 2015. Subjects: Social aspects, Technological innovations, Professional employees, Effect of technological innovations on...
The End of Lawyers?: Rethinking the Nature of Legal Services
First published 2010. Subjects: Practice of law, Technology and law, Lawyers, Effect of technological innovations on, Administration of J...
eBay marketplace picks
Marketplace Samples
Live-tested eBay searches with available results related to this page.
Selected fromlaw office decor oneBay.co.uk.
Endnotes
1.
Source: arxiv.org
Link:https://arxiv.org/abs/2405.20362
2.
Source: arxiv.org
Link:https://arxiv.org/abs/2401.01301
3.
Source: reuters.com
Title: us appeals court rebukes lawyer over fake hallucinated case citations 2026 07 10
Link:https://www.reuters.com/legal/litigation/us-appeals-court-rebukes-lawyer-over-fake-hallucinated-case-citations-2026-07-10/
Source snippet
Appeals Court has sharply criticized Florida attorney and county official Anthony Sabatini for submitting legal briefs that contained "fa...
4.
Source: americanbar.org
Title: American Bar Association AI Hallucinations Are Real—and How to Avoid Them
Link:https://www.americanbar.org/groups/litigation/resources/newsletters/appellate-practice/ai-hallucinations-real-how-avoid-them/
5.
Source: section1983.org
Title: mata v avianca inc
Link:https://www.section1983.org/cases/mata-v-avianca-inc/
Source snippet
Mata v. Avianca, Inc. — Case Law Library | section1983.org...
6.
Source: americanbar.org
Link:https://www.americanbar.org/groups/litigation/resources/newsletters/professional-liability/generative-ai-ethical-lawyer-standards-different-dilemmas-part-2/
Source snippet
American Bar AssociationGenAI and the Ethical Lawyer: Same Standards, Different Dilemmas Part II—How Courts Are Enforcing the Duty of Candor...
7.
Source: apnews.com
Link:https://apnews.com/article/c6a64736cb488cf6379624403d3757ca
Source snippet
U.S. District Judge Anna Manasco reprimanded William R. Lunsford, Matthew B. Reeves, and William J. Cranford for including unverified and...
8.
Source: americanbar.org
Link:https://www.americanbar.org/groups/litigation/resources/litigation-news/2026/fake-cases-real-sanctions-dangers-ai/
Additional References
9.
Source: layer3labs.io
Title: Lawyers Sanctioned for AI Hallucinations: Case List
Link:https://www.layer3labs.io/guides/lawyers-sanctioned-for-ai-hallucinations
Source snippet
July 15, 2026 — Reviewed by Jonathan West · Updated Jul 15, 2026 LAWYERS SANCTIONED FOR AI: A TRACKER OF FAKE-CITATION CASES Which lawyer...
Published: July 15, 2026
10.
Source: youtube.com
Title: AI and Ethics for Lawyers: Practical Guidance for Responsible Use
Link:https://www.youtube.com/watch?v=Z_dszp4uPPs
Source snippet
E170 | RealityCheck: BriefCatch's Front Against Hallucinations and Problems Beneath...
11.
Source: youtube.com
Link:https://www.youtube.com/watch?v=VGNYYee8Wy0
Source snippet
AI hallucinations in law: The cost of inaccuracy...
12.
Source: youtube.com
Title: AI, Liability, and Hallucinations in a Changing Tech and Law Environment
Link:https://www.youtube.com/watch?v=W9IOFKOdrSQ
Source snippet
AI and Ethics for Lawyers: Practical Guidance for Responsible Use...
13.
Source: dizon.law
Title: Karlo Dizon ·
Link:https://www.dizon.law/insights/ai-fabricated-citation-sanctions-cases
Source snippet
AI-Fabricated Citation Cases Every Litigator Should Know · DIZON.LAWJune 24, 2026 — AI-FABRICATED CITATION CASES EVERY LITIGATOR SHOULD K...
Published: June 24, 2026
14.
Source: youtube.com
Title: AI hallucinations in law: The cost of inaccuracy
Link:https://www.youtube.com/watch?v=JyChxtC8LVU
Source snippet
The Study That Proved AI Beats Lawyers? The AI Helped Grade Itself...
15.
Source: internationaltaxjournal.online
Link:https://internationaltaxjournal.online/index.php/itj/article/view/505
16.
Source: iclr.co.uk
Link:https://www.iclr.co.uk/blog/legal-profession/the-ai-accountability-reckoning-why-lawyers-cannot-delegate-professional-responsibility-to-algorithms/
17.
Source: airegulationhub.com
Title: Mata v. Avianca, Inc. — opinion and order on sanctions | Case Tracker
Link:https://www.airegulationhub.com/cases/d54e4b96-c528-4a75-a03c-bb25738057ab
18.
Source: legalaiinsights.com
Title: mata v avianca chatgpt citation hallucination sanctions 2023
Link:https://legalaiinsights.com/tool-evaluations/mata-v-avianca-chatgpt-citation-hallucination-sanctions-2023