Within Verification
Why AI's Best Work Can Still Fail
AI can be highly reliable on routine tasks yet fail unpredictably on small variations, making the most dangerous mistakes unusually hard to spot.
On this page
- What the jagged frontier means
- Why repeated success weakens human vigilance
- Which subtle errors cause the greatest harm
Page outline Jump by section
Introduction
One of the hardest problems in an AI-rich world is not that AI always fails. It is that it often succeeds so convincingly that its failures become difficult to notice. Modern AI systems can solve problems that challenge experienced professionals, yet make surprising mistakes on tasks that appear only slightly different. Researchers and practitioners increasingly describe this uneven pattern as the jagged frontier: AI capability does not advance smoothly but in irregular peaks and valleys.[cambrian.md]cambrian.mdWorking the jagged frontier: judgement as the core AI skillWorking the jagged frontier: judgement as the core AI skillJune 2, 2026…
This matters because the verification bottleneck is driven less by the number of errors than by their unpredictability. If every answer were unreliable, people would check everything. The real danger is that long runs of excellent performance encourage trust precisely when an unusual case falls outside the model’s hidden competence. As AI becomes a larger part of science, medicine, software engineering and public administration, understanding this mechanism becomes increasingly important for whether AI abundance translates into genuine human flourishing rather than an abundance of subtle mistakes.
What the jagged frontier actually means
People naturally imagine intelligence as a smooth scale: if a system can solve difficult problems, it should also master easier ones. Large language models repeatedly violate this intuition.
Instead of a steady increase in competence, they display a patchwork of strengths and weaknesses. A model may produce sophisticated software, explain advanced scientific concepts or pass professional examinations, then fail on a problem involving straightforward logical structure, unusual wording or an unfamiliar combination of concepts. The boundary between success and failure is therefore irregular rather than predictable.[cambrian.md]cambrian.mdWorking the jagged frontier: judgement as the core AI skillWorking the jagged frontier: judgement as the core AI skillJune 2, 2026…
This irregularity exists for several reasons:
- Models learn statistical relationships rather than explicit world models.
- Rare situations are less well represented in training data.
- Small changes in wording can push a problem into a region where the model generalises poorly.
- Improvements in one capability do not necessarily improve neighbouring capabilities.
The frontier is also dynamic. New model versions eliminate some weaknesses while introducing others, meaning organisations cannot assume that a workflow validated on one release will remain equally reliable after an update.[unfairstudio.ai]unfairstudio.aiUNFAI R Insights n°06 / The Jagged Frontier of AIUNFAI R Insights n°06 / The Jagged Frontier of AI
For organisations hoping AI will accelerate scientific discovery or make expertise more widely available, this creates a subtle challenge. Scaling intelligence is not enough if nobody can reliably identify the relatively rare cases where that intelligence quietly breaks down.
Why repeated success weakens human vigilance
The verification problem is as much psychological as technical.
Humans calibrate trust from experience. When an assistant is correct hundreds of times, checking every future answer feels increasingly unnecessary. This is a sensible strategy when dealing with tools whose failures are obvious and consistent. It becomes dangerous when failures are infrequent but highly consequential.
Researchers studying knowledge work found exactly this pattern. In a large field experiment involving hundreds of consultants, AI substantially improved performance on tasks that lay within its capability frontier. Yet on tasks outside that frontier, participants using AI often performed worse—and crucially, they struggled to recognise when they had crossed that invisible boundary.[Cambrian]cambrian.mdWorking the jagged frontier: judgement as the core AI skillWorking the jagged frontier: judgement as the core AI skillJune 2, 2026…
This produces several reinforcing effects.
Confidence transfers to nearby tasks. If AI has written excellent reports all week, users naturally expect similar quality on today’s report, even if today’s assignment contains unusual legal, medical or technical details.
Review becomes superficial. People tend to skim work they expect to be correct instead of actively searching for hidden flaws.
Automation bias develops. Psychological research has long shown that people often defer to automated recommendations, especially when the system has established a reputation for competence. AI’s conversational fluency can strengthen this tendency because errors are presented with the same confidence as correct answers.[AAAI Conference Proceedings]ojs.aaai.orgAAAI Conference ProceedingsConfirmation Bias: A Challenge for Scalable Oversight | Proceedings of the AAAI Conference on Artificial Intel…
The result is paradoxical. Better AI can sometimes make verification harder because its overall quality encourages less scrutiny while leaving a small number of difficult-to-detect failures.
Which subtle errors cause the greatest harm?
The most dangerous AI mistakes are rarely absurd. They usually look almost right.
Rather than producing obviously nonsensical answers, frontier models often generate outputs that are broadly plausible but contain one critical defect hidden within otherwise competent work.
Examples include:
- A legal analysis supported by fabricated case citations.
- A medical summary that omits a medication interaction while correctly describing everything else.
- Software that passes standard tests but leaves an uncommon security vulnerability unresolved.
- Financial reasoning built upon one incorrect assumption.
- Scientific summaries that accurately describe a field while introducing a few invented references.
These mistakes are difficult because reviewers cannot rely on intuition alone. The stronger the surrounding work, the easier it is for one incorrect statement to escape notice.
Recent research has highlighted another concern: benchmark optimisation itself can sometimes encourage confident guessing rather than calibrated uncertainty, increasing hallucinations even while headline accuracy scores improve. This illustrates why average benchmark performance alone may not capture practical reliability.[nature.com]nature.comApril 22, 2026…
In practice, organisations increasingly worry less about spectacular failures than about quiet defects that survive ordinary review.
Hidden errors often emerge at the edges
Many failures occur not because the entire task is difficult but because one exceptional condition changes the correct answer.
Examples include:
- Regulations containing unusual exemptions.
- Medical cases involving rare combinations of symptoms.
- Software behaviour under uncommon edge conditions.
- Contracts containing unusual jurisdiction-specific clauses.
- Scientific papers using specialised terminology outside common datasets.
These edge cases matter disproportionately because they are exactly where human expertise has traditionally added value.
Emerging research on rule-based reasoning has shown that nested exceptions and conditional logic remain challenging for many frontier models, with performance varying across model versions and even changing silently as providers update systems.[arXiv]arxiv.orgConfidently Wrong: Exception Chain Collapse in Frontier LLM Rule EvaluationJuly 25, 2026…
The implication is important for AI-enabled abundance. As AI takes over routine work, an increasing share of remaining human effort may concentrate on the exceptional cases where mistakes are hardest to predict.
Real-world reminders that plausible is not enough
Several recent incidents illustrate how hidden AI errors can escape normal quality control.
In 2025 and 2026, multiple consulting reports attributed to major firms were found to contain fabricated citations, invented references and unsupported claims despite appearing professionally produced. The reports looked polished enough that errors survived initial publication before external review identified them.[ft.com]ft.comThe firm stated it takes the issue seriously and is correcting faulty citations but did not clarify how the mistakes occurred. Other Big…
Software engineering provides another example. AI-generated code often works correctly and passes ordinary tests, yet security researchers continue to find cases where subtle vulnerabilities remain hidden because the visible functionality appears complete. Passing standard tests therefore does not necessarily imply that deeper verification has been achieved.[arXiv]arxiv.orgOpen source on arxiv.org.
These cases do not demonstrate that AI is unusable. They demonstrate that surface quality is an imperfect signal of underlying correctness.
Why this matters for an AI-enabled future
The optimistic vision of AI bloom depends on making intelligence far more abundant. If AI dramatically accelerates science, engineering, education and medicine, humanity could expand what it knows and what it can achieve over generations.
The jagged frontier does not invalidate that possibility. It identifies one of the practical constraints on reaching it.
If AI systems produce vastly more knowledge than humans can verify, then verification itself becomes the scarce resource. Progress will increasingly depend on building institutions, workflows and technical tools that identify the relatively small fraction of outputs requiring expert attention.
Several complementary approaches are already emerging:
- Retrieval systems that ground answers in verifiable documents rather than unsupported generation.
- Structured workflows that separate drafting from approval.
- Independent checking models that review outputs using different evidence.
- Domain-specific systems combining statistical AI with deterministic rules where correctness is essential.
- Better calibration, encouraging models to express uncertainty instead of confident guesses when evidence is weak.[nature.com]nature.comApril 22, 2026…
None of these eliminates the jagged frontier entirely. Instead, they aim to make its edges more visible.
For a future in which advanced AI contributes to broad human flourishing, the central challenge is not merely building systems that are usually correct. It is building societies that can recognise the rare moments when extraordinary competence quietly gives way to hidden error.
Amazon book picks
Further Reading
Books and field guides related to Why AI's Best Work Can Still Fail. Use these as the next step if you want deeper reading beyond the article.
Artificial Intelligence: A Guide for Thinking Humans
“After reading Mitchell’s guide, you’ll know what you don’t know and what other people don’t know, even though they claim to know it. And...
The Alignment Problem: Machine Learning and Human Values
Finalist for the Los Angeles Times Book Prize A jaw-dropping exploration of everything that goes wrong when we build AI systems and the m...
Co-Intelligence: Living and Working with AI
AN INSTANT NEW YORK TIMES BESTSELLER From Wharton professor and author of the popular One Useful Thing Substack newsletter Ethan Mollick...
Prediction Machines: The Simple Economics of Artificial Intel...
Cheap changes everything -- The magic of prediction -- Why it's called intelligence -- Data is the new oil -- The new division of labor -...
eBay marketplace picks
Marketplace Samples
Live-tested eBay searches with available results related to this page.
Selected fromAI robot art oneBay.co.uk.
Endnotes
1.
Source: cambrian.md
Title: Working the jagged frontier: judgement as the core AI skill
Link:https://www.cambrian.md/insights/working-the-jagged-frontier
Source snippet
Working the jagged frontier: judgement as the core AI skillJune 2, 2026...
Published: June 2, 2026
2.
Source: unfairstudio.ai
Title: UNFAI R Insights n°06 / The Jagged Frontier of AI
Link:https://www.unfairstudio.ai/en/insights/the-jagged-frontier-of-ai
3.
Source: ojs.aaai.org
Link:https://ojs.aaai.org/index.php/AAAI/article/view/41124
Source snippet
AAAI Conference ProceedingsConfirmation Bias: A Challenge for Scalable Oversight | Proceedings of the AAAI Conference on Artificial Intel...
4.
Source: arxiv.org
Link:https://arxiv.org/abs/2606.23491
5.
Source: nature.com
Link:https://www.nature.com/articles/s41586-026-10549-w
Source snippet
April 22, 2026...
Published: April 22, 2026
6.
Source: arxiv.org
Link:https://arxiv.org/abs/2607.23386
Source snippet
Confidently Wrong: Exception Chain Collapse in Frontier LLM Rule EvaluationJuly 25, 2026...
Published: July 25, 2026
7.
Source: ft.com
Link:https://www.ft.com/content/7e149ac8-2ce2-4266-8940-192f9821b33c
Source snippet
The firm stated it takes the issue seriously and is correcting faulty citations but did not clarify how the mistakes occurred. Other Big...
8.
Source: arxiv.org
Link:https://arxiv.org/abs/2605.17062
9.
Source: techradar.com
Link:https://www.techradar.com/pro/how-to-reduce-hallucinations-in-ai
Source snippet
To address these issues, a different approach called neurosymbolic AI is emerging. Developed by companies like UnlikelyAI, this hybrid me...
10.
Source: arxiv.org
Title: arXiv Steering the Verifiability of [Multimodal]({{ ‘multimodal-access/’ | relative_url }}) AI Hallucinations
Link:https://arxiv.org/abs/2604.06714
11.
Source: nature.com
Link:https://www.nature.com/articles/s41591-026-04501-8
12.
Source: nature.com
Link:https://www.nature.com/articles/s41586-026-10549-w_reference.pdf
13.
Source: nature.com
Link:https://www.nature.com/articles/s41586-026-10303-2
14.
Source: nature.com
Link:https://www.nature.com/articles/s41586-024-07421-0
15.
Source: envisioning.com
Link:https://www.envisioning.com/vocab/jagged-frontier
Additional References
16.
Source: axios.com
Link:https://www.axios.com/2025/04/23/ai-jagged-frontier-o3
Source snippet
While these models have received commendation for their advanced capabilities—such as enhanced vision and chain-of-thought reasoning thro...
17.
Source: youtube.com
Link:https://www.youtube.com/watch?v=mwC-aBlbQpY
Source snippet
Jagged technological frontier Ethan Mollick AI BCG study BCG tests GPT-4, the results will surprise you! (Harvard Study) AI Matej...
18.
Source: doi.org
Link:https://doi.org/10.1038/s41586-026-10549-w
Source snippet
Evaluating large language models for accuracy incentivizes hallucinations | Nature...
19.
Source: youtube.com
Link:https://www.youtube.com/watch?v=FR3mvwP-VcA
Source snippet
Does Generative AI Really Help Skilled Workers? MIT Study Reveals a 40% Boost—Or a 19% Drop...
20.
Source: youtube.com
Title: How to Use Generative AI in Consulting (with HBS And BCG) | CXOTalk #820
Link:https://www.youtube.com/watch?v=ZhWlGqkLC_M
Source snippet
Artificial Jagged Intelligence: Why AI is Brilliantly Smart and Completely Stupid at the Same Time...
21.
Source: youtube.com
Title: Navigating the jagged frontier of generative AI | Ethan Mollick and Lonne Jaffe
Link:https://www.youtube.com/watch?v=Yka2S-yTX70
Source snippet
How to Use Generative AI in Consulting (with HBS And BCG) | CXOTalk #820...
22.
Source: youtube.com
Title: Ethan Mollick: “Navigating the Jagged Technological Frontier”
Link:https://www.youtube.com/watch?v=dPJ6Bxsky0s
Source snippet
Navigating the jagged frontier of generative AI | Ethan Mollick and Lonne Jaffe...
23.
Source: reddit.com
Link:https://www.reddit.com/r/LLMDevs/comments/1tuk7jl/i_tested_5_frontier_llms_on_fixing_realworld/
24.
Source: aclanthology.org
Link:https://aclanthology.org/2026.findings-eacl.206/
25.
Source: fenado.ai
Link:https://fenado.ai/articles/mollick-pinpoints-uncharted-jaggedness-in-organizational-multi-agent-ai-workflows