Safety Gaps

Introduction

AI agents are increasingly marketed as systems that can plan, act, browse, code, purchase, coordinate workflows, and complete multi-step tasks with limited human supervision. Yet the public often learns remarkably little about how these systems are tested, constrained, monitored, or shut down when things go wrong. That disclosure gap matters because agentic AI changes the nature of AI risk. A chatbot that produces bad advice is one problem. An autonomous system with access to software tools, databases, cloud infrastructure, payment systems, or internal company workflows creates a very different challenge.

Safety Gaps illustration 1 The concern is not only immediate consumer harm. The broader AI bloom vision depends on society eventually trusting advanced systems with increasingly important scientific, industrial, medical, and civilisational tasks. If AI is to help accelerate discovery, coordinate complex infrastructure, or expand human capability safely over decades, institutions will need much stronger evidence about how autonomous systems behave under stress, misuse, failure, or adversarial conditions. Right now, public disclosures are often too thin to support that confidence. [arXiv]arxiv.orgarXivThe 2025 AI Agent Index Documenting Technical and…19 Feb 2026 — We present findings from the 2025 AI Agent Index across six categ… [International]arxiv.orgInternational AI Safety Report 2025: First Key Updateby Y Bengio · 2025 · Cited by 5 — Abstract:Since the publication of the first Intern…

What the AI Agent Index says is missing

One of the clearest recent attempts to measure this problem is the 2025 AI Agent Index, a research project examining 30 widely deployed AI agents across categories including autonomy, ecosystem interaction, governance, and safety documentation. Its findings were striking: only a small minority of agents had published formal safety and evaluation documents specifically addressing the deployed agent itself rather than the underlying language model. arXiv [AI Agent Index]aiagentindex.mit.edu2025 AI Agent IndexAI Agent IndexThe 2025 AI Agent Index: Documenting Technical and Safety…by L STAUFER · Cited by 1 — We present findings from the 2025…

The gap matters because many companies now market “agentic” systems aggressively while disclosing very little about:

what environments the agent was tested in,
how often it fails,
what permissions it receives,
how much human oversight exists,
whether independent audits occurred,
how shutdown mechanisms work,
or how the system behaves after deployment updates.

The Index argues that the ecosystem is “inconsistently documented”, making meaningful comparison difficult for regulators, researchers, businesses, and ordinary users. [arXiv]arxiv.orgarXivThe 2025 AI Agent Index Documenting Technical and…19 Feb 2026 — We present findings from the 2025 AI Agent Index across six categ…

This is not merely a paperwork issue. Safety disclosure is one of the few ways outsiders can distinguish between a genuinely constrained system and a marketing narrative. An AI agent may advertise itself as “human-supervised”, for example, while the actual supervision consists only of occasional spot checks after actions have already occurred. Another system may claim strong safeguards without revealing whether those safeguards were tested against prompt injection attacks, cascading tool failures, or attempts to bypass restrictions.

The result is a widening asymmetry between capability claims and accountability evidence. Companies frequently demonstrate what agents can do in polished demos, but provide far less information about failure rates, unsafe edge cases, or operational limits.

Why agents need different evaluations from chatbots

Many existing AI safety practices were designed around conversational systems rather than autonomous actors. That distinction is becoming increasingly important.

A traditional chatbot mainly generates outputs for a human to review. An agent instead combines reasoning with execution. It may:

access tools,
operate continuously,
maintain memory,
trigger software actions,
communicate with other systems, [researchgate.net]researchgate.netThe 2025 AI Agent Index: Documenting Technical and…19 Feb 2026 — Agentic AI systems are increasingly capable of performing professiona…
or revise plans dynamically.

That creates risks which do not appear clearly in ordinary benchmark testing. [International AI Safety Report]internationalaisafetyreport.orgInternational AI Safety Report2026 Report: Extended Summary for PolicymakersFeb 3, 2026 — AI agents can increase reliability risks by car… [International]arxiv.orgInternational AI Safety Report 2025: First Key Updateby Y Bengio · 2025 · Cited by 5 — Abstract:Since the publication of the first Intern…

An agent can appear safe in isolated laboratory evaluations yet behave differently once connected to live environments. Problems emerge from interaction loops rather than single outputs. Researchers increasingly warn about “goal-plan-execution gaps”, where systems interpret broad objectives unpredictably during real-world execution. [SSRN]papers.ssrn.comThis paper explains why by highlighting three sociotechnical challenges…

This is why many researchers argue that disclosure should include operational information rather than only model-level metrics. Useful disclosures for agentic systems would include:

escalation procedures,
permission boundaries,
rollback mechanisms,
audit logging,
maximum action scope,
rate limits,
human approval checkpoints,
and post-deployment monitoring practices.

Without these details, outsiders cannot evaluate whether an agent is genuinely constrained or simply trusted not to fail.

The International AI Safety Report repeatedly stresses that agentic systems reduce opportunities for direct human intervention and create distinctive reliability risks because humans are no longer reviewing every step manually. [International AI Safety Report]internationalaisafetyreport.orgInternational AI Safety Report2026 Report: Extended Summary for PolicymakersFeb 3, 2026 — AI agents can increase reliability risks by car… [International]arxiv.orgInternational AI Safety Report 2025: First Key Updateby Y Bengio · 2025 · Cited by 5 — Abstract:Since the publication of the first Intern…

That changes the burden of proof. A company deploying a model that merely drafts emails may reasonably disclose less than a company deploying an agent capable of writing code, managing infrastructure, or autonomously interacting with external systems. Yet disclosure standards often remain similar even as system autonomy expands.

The missing information that matters most

The current disclosure problem is not only about secrecy. It is also about selective visibility. Many companies publish high-level principles while omitting operational specifics that would allow outsiders to evaluate actual risk management.

Several recurring gaps stand out.

Pre-deployment versus post-deployment testing

Many public evaluations focus on systems before release. Far less information is available about how agents behave after deployment, updates, tool integrations, or real-world adaptation.

A recent paper on frontier AI evaluations argued that companies should disclose both pre-mitigation and post-mitigation testing results because otherwise policymakers and users cannot judge whether safeguards genuinely reduced dangerous behaviours or merely shifted them elsewhere. [arXiv]arxiv.orgarXivThe 2025 AI Agent Index Documenting Technical and…19 Feb 2026 — We present findings from the 2025 AI Agent Index across six categ…

This distinction matters especially for agents because deployment environments are highly dynamic. A harmless-looking planning system may become substantially more powerful once connected to browsers, APIs, databases, payment tools, robotics, or cloud services.

External evaluator access

Another major problem is limited outside scrutiny.

Researchers examining frontier AI evaluation practices argue that external evaluators often receive insufficient access, incomplete documentation, and little time to test advanced systems rigorously. [arXiv]arxiv.orgarXivThe 2025 AI Agent Index Documenting Technical and…19 Feb 2026 — We present findings from the 2025 AI Agent Index across six categ…

In practice, many outside audits resemble constrained demonstrations rather than adversarial investigations. Evaluators may lack access to:

internal system prompts,
tool permissions,
hidden orchestration layers,
deployment logs,
or long-duration testing environments.

That weakens confidence in safety claims, particularly when commercial incentives favour rapid deployment.

Internal deployments

One of the least transparent areas concerns internal AI deployments inside frontier labs themselves.

Advanced companies increasingly use AI systems to automate software engineering, research assistance, security analysis, and model development. Yet there is little public visibility into how these internal agents are governed. [arXiv]arxiv.orgarXivThe 2025 AI Agent Index Documenting Technical and…19 Feb 2026 — We present findings from the 2025 AI Agent Index across six categ…

That matters because internal deployments can expose systems to sensitive infrastructure long before the public sees them. If a company eventually hopes to build highly autonomous research agents capable of accelerating science and economic productivity, internal experimentation may become one of the earliest real tests of large-scale machine autonomy.

The disclosure problem therefore affects not only public consumer tools but also the developmental path toward more capable future systems.

Safety Gaps illustration 2

Why weak disclosures undermine the broader AI bloom case

Supporters of long-term AI optimism often argue that advanced agents could eventually help civilisation flourish at unprecedented scale. In the strongest versions of the AI bloom argument, autonomous systems could accelerate medicine, coordinate infrastructure, reduce dangerous labour, improve scientific discovery, and expand humanity’s long-term capabilities.

But that optimistic future depends heavily on institutional trust.

A society will not hand increasingly important infrastructure to advanced AI systems merely because they are economically useful. It will require confidence that failures can be detected, understood, contained, and corrected. Thin disclosure weakens that confidence in several ways.

First, it makes independent verification difficult. Outsiders cannot distinguish robust safety practices from public relations language if companies disclose only vague principles.

Second, it slows institutional learning. Aviation, medicine, nuclear safety, and cybersecurity all improved partly because incidents, near misses, standards, and testing methods became increasingly visible across industries. Weak disclosure fragments knowledge and prevents collective learning.

Third, it increases the risk of reactive governance. If the public repeatedly discovers hidden failures after deployment rather than before, political systems may respond with blunt restrictions instead of targeted oversight frameworks.

That tension increasingly appears in frontier AI governance debates. Several recent analyses argue that current frontier safety policies focus heavily on prevention narratives while underinvesting in coordination, transparency, and ecosystem-wide resilience when failures occur. [ResearchGate]researchgate.netThe 2025 AI Agent Index: Documenting Technical and…19 Feb 2026 — Agentic AI systems are increasingly capable of performing professiona…

For the AI bloom thesis, this matters deeply. The long-term promise of AI depends not only on capability growth but on whether advanced systems become governable enough for broad, durable social legitimacy.

Questions users and regulators should ask before deployment

As AI agents move from demonstrations into workplaces, infrastructure, research environments, and consumer software, disclosure quality becomes a practical governance issue rather than an abstract ethics debate.

Several questions increasingly separate meaningful transparency from superficial assurance.

What exactly can the agent do autonomously?

Companies often describe systems as “assistants” even when they can independently execute multi-step workflows. Useful disclosure should clarify:

whether the system can take actions without approval,
how long it can operate continuously,
what external tools it can access,
and what financial, technical, or operational authority it possesses.

Where are the human checkpoints?

“Human in the loop” can mean many different things.

A meaningful checkpoint may require approval before high-impact actions. A weak checkpoint may merely allow humans to review logs after actions are completed. Public disclosures rarely explain this distinction clearly.

Safety Gaps illustration 3

What failure rates were observed?

Many AI demonstrations highlight successful task completion while revealing little about reliability under realistic conditions.

For agents, useful disclosure would include:

task failure rates,
hallucination rates during tool use,
unsafe action frequencies,
escalation failures,
and known operational limitations.

Can the system be audited independently?

Independent evaluation is difficult if external researchers lack meaningful access to the deployed environment. Questions about evaluator access, audit scope, and reproducibility are increasingly central. [arXiv]arxiv.orgarXivThe 2025 AI Agent Index Documenting Technical and…19 Feb 2026 — We present findings from the 2025 AI Agent Index across six categ…

What happens if the agent behaves unexpectedly?

Shutdown and containment procedures are especially important for systems capable of persistence and autonomous action.

Yet disclosures often provide almost no operational detail about:

rollback mechanisms,
containment architecture,
privilege separation,
or emergency intervention protocols.

This absence becomes more concerning as systems gain broader real-world access.

The deeper governance problem behind thin disclosures

Part of the disclosure gap is commercial. Companies fear revealing competitive information, exposing vulnerabilities, or increasing liability.

But another problem is structural: governance systems have not yet adapted to software that behaves less like a static product and more like a semi-autonomous participant in economic and technical systems.

Traditional software audits assumed relatively deterministic behaviour. Agentic systems are probabilistic, adaptive, and highly context-dependent. Evaluating them thoroughly is expensive, technically difficult, and continuously changing as models update.

That creates incentives to disclose broad principles while avoiding measurable commitments.

Meanwhile, capability races intensify pressure to ship quickly. The AI Agent Index researchers note that deployment is accelerating faster than standardisation around accountability and documentation. [arXiv]arxiv.orgarXivThe 2025 AI Agent Index Documenting Technical and…19 Feb 2026 — We present findings from the 2025 AI Agent Index across six categ…

This does not mean advanced AI governance is impossible. Other high-risk industries eventually developed layered systems of audits, reporting standards, incident disclosure, certification, and operational oversight. But those systems often emerged only after repeated failures forced institutional adaptation.

The central question for agentic AI is whether governance mechanisms can mature before autonomy scales much further.

Why this issue grows more important as AI capability increases

Current AI agents remain error-prone and limited in many domains. The International AI Safety Report notes that today’s systems still fail basic tasks and are not yet fully autonomous in the strongest sense. [International AI Safety Report]internationalaisafetyreport.orgInternational AI Safety Report2026 Report: Extended Summary for PolicymakersFeb 3, 2026 — AI agents can increase reliability risks by car…

But the disclosure problem may become more serious precisely because capabilities are improving incrementally rather than explosively.

As systems become more reliable, organisations will naturally grant them more authority. Agents that save time in low-risk workflows today may later receive broader permissions across finance, infrastructure, logistics, cybersecurity, scientific research, or autonomous laboratories.

In the optimistic AI bloom scenario, this expanding delegation could eventually unlock enormous gains in productivity, discovery, and human flourishing. But those gains depend on society being able to evaluate and trust increasingly autonomous systems.

Thin disclosures therefore create a paradox. The more important AI agents become to the future economy and to humanity’s long-term trajectory, the less acceptable opaque governance becomes.

The challenge is not only building capable agents. It is building institutions capable of proving that those agents remain understandable, governable, interruptible, and aligned with human purposes as their autonomy expands.

Endnotes

Source: arxiv.org
Link: https://arxiv.org/html/2602.17753v1
Source snippet
arXivThe 2025 AI Agent Index Documenting Technical and...19 Feb 2026 — We present findings from the 2025 AI Agent Index across six categ...
Source: researchgate.net
Link: https://www.researchgate.net/publication/401057786_The_2025_AI_Agent_Index_Documenting_Technical_and_Safety_Features_of_Deployed_Agentic_AI_Systems
Source snippet
The 2025 AI Agent Index: Documenting Technical and...19 Feb 2026 — Agentic AI systems are increasingly capable of performing professiona...
Source: papers.ssrn.com
Link: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5529058
Source snippet
This paper explains why by highlighting three sociotechnical challenges...
Source: arxiv.org
Link: https://arxiv.org/abs/2503.17388
Source snippet
arXivAI Companies Should Report Pre- and Post-Mitigation...by D Bowen · 2025 · Cited by 2 — In this position paper, we argue that fronti...
Source: arxiv.org
Link: https://arxiv.org/abs/2601.11916
Source snippet
arXivExpanding External Access To Frontier AI Models For Dangerous Capability EvaluationsJanuary 17, 2026...

Published: January 17, 2026
Source: arxiv.org
Title: arXiv What Should Frontier AI Developers Disclose About Internal Deployments?
Link: https://arxiv.org/abs/2604.23065
Source snippet
arXivWhat Should Frontier AI Developers Disclose About Internal Deployments?April 25, 2026...

Published: April 25, 2026
Source: researchgate.net
Title: Research Gate(PDF) The Coordination Gap in Frontier AI Safety Policies
Link: https://www.researchgate.net/publication/401076243_The_Coordination_Gap_in_Frontier_AI_Safety_Policies
Source snippet
ResearchGate(PDF) The Coordination Gap in Frontier AI Safety PoliciesFebruary 21, 2026 — 21 Feb 2026 — Frontier AI Safety Policies concen...

Published: February 21, 2026
Source: arxiv.org
Title: arXiv The coordination gap in frontier AI safety policies
Link: https://arxiv.org/abs/2603.10015
Source: researchgate.net
Title: 401178467 International AI Safety Report 2026
Link: https://www.researchgate.net/publication/401178467_International_AI_Safety_Report_2026
Source snippet
(PDF) International AI Safety Report 2026The International AI Safety Report 2026 synthesises the current scientific evidence on the capab...
Source: researchgate.net
Title: 399787839 Trustworthy Agentic AI Balancing Autonomy with Human Oversight
Link: https://www.researchgate.net/publication/399787839_Trustworthy_Agentic_AI_Balancing_Autonomy_with_Human_Oversight
Source snippet
Trustworthy Agentic AI: Balancing Autonomy with Human...Mar 3, 2026 — This paper proposes a practical pattern for trustworthy agentic AI...
Source: arxiv.org
Link: https://arxiv.org/abs/2602.21012
Source snippet
[2602.21012] International AI Safety Report 2026by Y Bengio · 2026 · Cited by 56 — The International AI Safety Report 2026 synthesises th...
Source: arxiv.org
Link: https://arxiv.org/abs/2510.13653
Source snippet
International AI Safety Report 2025: First Key Updateby Y Bengio · 2025 · Cited by 5 — Abstract:Since the publication of the first Intern...
Source: arxiv.org
Link: https://arxiv.org/pdf/2602.21012
Source snippet
International AI Safety Report 2026by Y Bengio · 2026 · Cited by 57 — The Panel comprises representatives nominated by over 30 countries...
Source: arxiv.org
Link: https://arxiv.org/abs/2501.17805
Source snippet
[2501.17805] International AI Safety Reportby Y Bengio · 2025 · Cited by 146 — The first International AI Safety Report comprehensively s...
Source: internationalaisafetyreport.org
Link: https://internationalaisafetyreport.org/publication/international-ai-safety-report-2025
Source snippet
International AI Safety ReportInternational AI Safety Report 2025Jan 29, 2025 — The purpose of this report is to help create a shared int...
Source: internationalaisafetyreport.org
Link: https://internationalaisafetyreport.org/publication/2026-report-extended-summary-policymakers
Source snippet
International AI Safety Report2026 Report: Extended Summary for PolicymakersFeb 3, 2026 — AI agents can increase reliability risks by car...
Source: aiagentindex.mit.edu
Title: 2025 AI Agent Index
Link: https://aiagentindex.mit.edu/data/2025-AI-Agent-Index.pdf
Source snippet
AI Agent IndexThe 2025 AI Agent Index: Documenting Technical and Safety...by L STAUFER · Cited by 1 — We present findings from the 2025...
Source: cam.ac.uk
Title: ai agent index safety
Link: https://www.cam.ac.uk/stories/ai-agent-index-safety
Source snippet
University of CambridgeMost AI bots lack basic safety disclosures, study finds20 Feb 2026 — An investigation into 30 top AI agents finds...
Source: internationalaisafetyreport.org
Title: international ai safety report 2026
Link: https://internationalaisafetyreport.org/publication/international-ai-safety-report-2026
Source snippet
International AI Safety ReportInternational AI Safety Report 2026Feb 3, 2026 — In addition, AI agents – systems that can act in the world...
Source: insideprivacy.com
Link: https://www.insideprivacy.com/artificial-intelligence/international-ai-safety-report-2026-examines-ai-capabilities-risks-and-safeguards/
Source snippet
International AI Safety Report 2026 Examines AI...12 Feb 2026 — According to the Report, such scenarios may occur if systems develop the...
Source: globalpolicywatch.com
Link: https://www.globalpolicywatch.com/2026/02/international-ai-safety-report-2026-examines-ai-capabilities-risks-and-safeguards/
Source snippet
fraud, blackmail, extortion, defamation, and the production of non... Researchers are also developing detection tools that work even in...
Source: internationalaisafetyreport.org
Title: international ai safety report 2026
Link: https://internationalaisafetyreport.org/sites/default/files/2026-02/international-ai-safety-report-2026.pdf
Source snippet
2026Feb 1, 2026 — This Report is a synthesis of the existing research on the capabilities and risks of advanced AI. The Report does not n...
Source: internationalaisafetyreport.org
Title: International AI Safety Report The Executive
Link: https://internationalaisafetyreport.org/
Source snippet
International AI Safety ReportThe Executive Summary offers a concise three-page overview of the 2026 Report's core findings on general-pu...
Source: linkedin.com
Title: international ai safety report 2025 key insights stefano besana zb7zf
Link: https://www.linkedin.com/pulse/international-ai-safety-report-2025-key-insights-stefano-besana-zb7zf
Source snippet
The International AI Safety Report 2025: Key Insights and...The report categorizes AI risks into three primary areas: Malicious Use Risk...
Source: aigl.blog
Title: international ai safety report first key update october 2025
Link: https://www.aigl.blog/international-ai-safety-report-first-key-update-october-2025/
Source snippet
International AI Safety Report: First Key Update (October...Nov 7, 2025 — A concise “key update” on fast-moving frontier AI: reasoning m...

Published: october 2025
Source: lumenova.ai
Title: ai agent index aiai
Link: https://www.lumenova.ai/blog/ai-agent-index-aiai/
Source snippet
What You Should Know: The AI Agent Index15 Apr 2025 — Find out how the AI Agent Index (AIAI) documents AI systems, tracks risks & reveals...

Additional References

Source: linkedin.com
Link: https://www.linkedin.com/posts/carina-prunkl-688a4795_big-day-the-2026-international-ai-safety-activity-7424464214825635841-H4lH
Source snippet
2026 International AI Safety Report HighlightsBig day: the 2026 International AI Safety Report is out! It was a privilege to serve as Lea...
Source: mckinsey.com
Title: deploying agentic ai with safety and security a playbook for technology leaders
Link: https://www.mckinsey.com/capabilities/risk-and-resilience/our-insights/deploying-agentic-ai-with-safety-and-security-a-playbook-for-technology-leaders
Source snippet
Agentic AI security: Risks & governance for enterprisesOct 16, 2025 — Explore agentic AI security best practices, including AI governance...
Source: aigl.blog
Title: principles for evaluating misuse safeguards of frontier ai systems
Link: https://www.aigl.blog/principles-for-evaluating-misuse-safeguards-of-frontier-ai-systems/
Source snippet
Principles for Evaluating Misuse Safeguards of Frontier AI...3 Apr 2025 — This guidance lays out a concrete plan for assessing whether s...
Source: complexdiscovery.com
Title: AI agents are operating with progressively less human oversight
Link: https://complexdiscovery.com/2026-ai-safety-report-flags-escalating-threats-for-cyber-ig-and-ediscovery-professionals/
Source snippet
2026 AI Safety Report Flags Escalating Threats for Cyber...Feb 11, 2026 — The 2026 International AI Safety Report reveals AI-driven cyb...
Source: rand.org
Link: https://www.rand.org/content/dam/rand/pubs/conf_proceedings/CFA3400/CFA3429-1/RAND_CFA3429-1.pdf
Source snippet
ng AI systems' capabilities, applicable across varied risk scenarios, to ensure that...Read more...
Source: techcommunity.[microsoft]({{ ‘ai-bloom-abun/ai-bloom-abun-98d3a6-shared-ai-gai-89312d-ai-platform-l-e1f9a1-microsoft-ope-846027/’ | relative_url }}). com
Title: security as the core primitive securing ai agents and apps
Link: https://techcommunity.microsoft.com/blog/microsoft-security-blog/security-as-the-core-primitive—securing-ai-agents-and-apps/4470197
Source snippet
as the core primitive - Securing AI agents and apps18 Nov 2025 — These are: preventing agent sprawl and access to resources, protecting a...
Source: beam.ai
Title: how to audit ai agents before enterprise security review
Link: https://beam.ai/agentic-insights/how-to-audit-ai-agents-before-enterprise-security-review
Source snippet
How to Audit AI Agents Before a Security Review in 20268 days ago — 88% of orgs running AI agents had a security incident. The 8-point pr...
Source: ibm.com
Title: new global ai safety report means enterprise
Link: https://www.ibm.com/think/news/new-global-ai-safety-report-means-enterprise
Source snippet
What a new global AI safety report means for enterpriseFeb 23, 2026 — According to the 2026 International AI Safety Report, the most pres...
Source: loginradius.com
Title: auditing and logging ai agent activity
Link: https://www.loginradius.com/blog/engineering/auditing-and-logging-ai-agent-activity
Source snippet
A Guide for Engineers25 Feb 2026 — Agentic AI security frameworks must ensure that delegation logs are immutable, time-stamped, and crypt...
Source: fin.ai
Title: evaluate ai agent security compliance
Link: https://fin.ai/learn/evaluate-ai-agent-security-compliance
Source snippet
AI Agent Security Evaluation: Compliance Guide13 Mar 2026 — This guide provides a structured approach for evaluating AI customer service...