Human review

Introduction

The safest and most effective public-sector AI systems usually do not replace officials. They help officials cope with complexity, volume, and urgency while leaving accountable humans responsible for final judgement. That distinction matters because governments exercise powers that directly affect liberty, income, healthcare, education, immigration status, and disaster relief. When mistakes happen, citizens need to know who is responsible, how to appeal, and whether unusual circumstances can still be recognised.

Human review illustration 1 This is one of the clearest lessons from early government AI deployments. Systems that narrow attention, highlight anomalies, summarise evidence, or prioritise urgent cases often improve state capacity without removing democratic accountability. Systems that silently automate high-stakes decisions have repeatedly produced backlash, legal challenges, or public distrust. The debate is therefore not simply whether governments should use AI, but where human judgement must remain central if an AI-enabled state is to become more capable without becoming less legitimate. [Open Government Partnership]opengovpartnership.orgOpen Government PartnershipAlgorithmic accountability for the public sectorThis executive summary highlights the key findings from this s…

Why assistance often works better than automation

Government decisions differ from many commercial decisions because they involve legal rights, public obligations, and edge cases that cannot always be reduced to historical patterns. A mistaken recommendation on a shopping website is inconvenient. A mistaken benefits suspension or immigration refusal can destabilise a life.

For that reason, many successful public-sector systems act as decision-support tools rather than autonomous decision-makers. The AI helps officials process large amounts of information quickly, but humans remain responsible for interpretation, exceptions, and accountability.

Several practical reasons explain why this model tends to work better.

Public rules contain exceptions. Welfare systems, disaster aid, healthcare access, and tax administration all involve unusual circumstances that may not appear clearly in training data.
Governments must justify decisions. Citizens often have legal rights to explanations, review, and appeal.
Data is incomplete or historically biased. Past records may reflect discrimination, unequal enforcement, or administrative blind spots.
Officials manage competing values. Efficiency is only one goal alongside fairness, legality, proportionality, and public trust.
Crises change conditions rapidly. Models trained on ordinary conditions can fail during floods, pandemics, or economic shocks.

Research on “human-AI interaction” in government also shows that officials can become overly reliant on algorithmic recommendations even when warning signs exist, a phenomenon known as automation bias. At the same time, people may selectively trust algorithmic outputs when they reinforce existing assumptions or stereotypes. In practice, this means that simply adding a human reviewer is not automatically enough; institutions must design workflows that preserve meaningful judgement rather than rubber-stamping machine outputs. [arXiv]arxiv.orgarXiv Algorithms and Decision-Making in the Public SectorarXiv Algorithms and Decision-Making in the Public Sector [OUP]academic.oup.comArtificial intelligence algorithms are increasingly adopted as decisional aides by public bodies, with the promise of overcoming biases o…

Disaster response: narrowing attention without pretending to understand everything

Disaster response shows where AI assistance can genuinely expand state capability without displacing human judgement.

After floods, earthquakes, storms, or wildfires, governments face an information overload problem. Satellite imagery, drone footage, emergency calls, infrastructure data, weather forecasts, and local reports arrive faster than human teams can process them. Computer vision systems can rapidly identify likely damaged buildings, blocked roads, or flooded areas, allowing officials to prioritise inspections and dispatch aid more quickly.

But the AI normally does not decide who receives assistance. Human teams still evaluate conditions on the ground, verify local context, and weigh competing priorities.

That division of labour matters because disasters contain exactly the kinds of situations algorithms struggle to interpret:

elderly residents refusing evacuation;
undocumented residents afraid to contact authorities;
damaged communities with poor digital records;
conflicting reports from local agencies;
infrastructure failures that distort incoming data.

In these environments, AI improves coordination by compressing the search space. Officials no longer need to manually scan every image or document. But humans remain essential because disaster response involves moral and political judgement as much as pattern recognition.

This distinction also hints at a larger possibility within the broader AI bloom vision. Advanced AI could eventually help societies coordinate far more intelligently during crises, improving resilience against climate shocks, pandemics, or infrastructure failures. Yet the public legitimacy of those systems will depend partly on whether citizens believe humans still retain responsibility for consequential decisions.

Tax, welfare, and healthcare administration

The same “assist rather than decide” model appears across administrative systems.

Tax agencies increasingly use machine learning to flag unusual transactions, detect possible fraud patterns, or prioritise audits. Healthcare systems use AI to identify potentially urgent claims, summarise records, or detect anomalies in billing data. Welfare departments use algorithms to identify cases that may require further review.

These systems can produce genuine gains:

faster processing times;
reduced administrative backlog;
earlier detection of fraud; [ohchr.org]ohchr.orglandmark ruling dutch court stops government attempts spy poor un expertOHCHRLandmark ruling by Dutch court stops government attempts…5 Feb 2020 — The court ordered the immediate halt to a digital benefit f…
more targeted inspections;
lower bureaucratic workload;
quicker identification of urgent cases.

But they become far more controversial when automated outputs directly determine outcomes without meaningful review.

The Netherlands’ SyRI welfare fraud system became a major warning example. SyRI combined data from multiple government databases to generate fraud-risk indicators for welfare recipients. Critics argued that the system lacked transparency and disproportionately targeted poorer communities. In 2020, a Dutch court ruled that the system violated human rights protections, particularly privacy rights under the European Convention on Human Rights. The court highlighted problems including opacity, insufficient safeguards, and imbalance between state surveillance powers and citizen protections. [The Library of Congress]loc.govThe Library of CongressNetherlands: Court Prohibits Government's Use of AI…The SyRI system is an algorithm used by the government to p… [OHCHR]ohchr.orglandmark ruling dutch court stops government attempts spy poor un expertOHCHRLandmark ruling by Dutch court stops government attempts…5 Feb 2020 — The court ordered the immediate halt to a digital benefit f… [Rechtspraak The issue was not merely technical inaccuracy. It was institutional legitimacy. Citizens could not properly understand]rechtspraak.nlRechtspraakSyRI legislation in breach of European Convention on…The Hague District Court has delivered a judgment today in a case abou…, challenge, or inspect the logic affecting them.

The UK’s 2020 A-level grading controversy exposed similar problems from another direction. During the pandemic, an algorithmic standardisation system downgraded many students based partly on historical school performance patterns. The backlash was intense because individual students felt trapped inside statistical averages that ignored personal circumstances and achievement. The government eventually abandoned the system and reverted to teacher assessments. [WIRED]wired.comThe algorithm, designed by exams regulator Ofqual, aimed to standardize results nationally but ended up downgrading 40% of predicted grad… [LSE Blogs]blogs.lse.ac.ukfk the algorithm what the world can learn from the uks a level grading fiascoThis is a well-established problem that data professionals have sought to…Read more… [Global Government Forum]globalgovernmentforum.comuk government drops exam grading algorithm in the face of public angerUK government drops exam grading algorithm in the face…18 Aug 2020 — The UK government has been forced to overturn exam results produc…

Again, the problem was not that statistical models are always useless. Large-scale systems inevitably require statistical reasoning. The deeper problem was allowing aggregate modelling logic to override case-level human judgement in decisions that strongly shaped individual futures.

Why human review matters for rights and appeals

Human review is often discussed as a technical safeguard, but it is also a constitutional principle.

Modern states derive legitimacy partly from the idea that power can be questioned, appealed, and corrected. When decisions become difficult to explain or contest, trust deteriorates even if the system improves average efficiency.

Human involvement matters most in areas where decisions affect:

legal status;
income security;
education access;
healthcare access;
policing or sentencing;
immigration outcomes;
child protection;
housing rights.

In these domains, citizens usually expect several things simultaneously:

Human review illustration 2

A decision-maker who can understand context.
A process for correcting mistakes.
A reason that can be explained in ordinary language.
Accountability when harms occur.

Purely automated systems struggle with all four.

This is one reason European regulation increasingly treats many government AI applications as “high-risk” systems requiring oversight, documentation, auditability, and human supervision. Emerging debates around the EU AI Act focus heavily on preserving administrative law principles such as proportionality, explainability, and reviewability in public-sector deployments. [arXiv]arxiv.orgarXiv Algorithms and Decision-Making in the Public SectorarXiv Algorithms and Decision-Making in the Public Sector

Importantly, meaningful human review is not the same as superficial approval. If an official simply clicks “confirm” on machine recommendations all day under productivity pressure, accountability may become performative rather than real.

Where assistance quietly turns into hidden automation

One of the most important public-sector risks is not fully autonomous AI. It is hidden automation: systems that officially preserve human oversight while practically steering decisions so strongly that humans rarely disagree.

This can happen gradually.

An agency introduces an AI system only to “assist” staff. Officials are initially encouraged to use their own judgement. Over time:

staffing levels shrink because the system appears efficient;
case quotas increase;
managers monitor whether staff follow model recommendations;
overturning the AI requires extra paperwork;
reviewers become fatigued;
unusual cases receive less time.

Eventually the AI recommendation becomes the de facto decision even if a human technically remains involved.

Researchers studying public-sector algorithms warn that this dynamic can create “automation bias”, where officials defer excessively to algorithmic outputs. Citizens may still believe a real human evaluated their case individually even when practical discretion has largely disappeared. [arXiv]arxiv.orgarXiv Algorithms and Decision-Making in the Public SectorarXiv Algorithms and Decision-Making in the Public Sector [OUP]academic.oup.comArtificial intelligence algorithms are increasingly adopted as decisional aides by public bodies, with the promise of overcoming biases o…

This matters because public institutions are often resource constrained. AI systems are frequently introduced during periods of budget pressure or administrative overload. The temptation to quietly convert decision-support systems into labour-reduction systems can become very strong.

The UK debate around welfare fraud algorithms illustrates this tension. Investigations into Department for Work and Pensions systems found large numbers of false positives, with many flagged cases ultimately proving legitimate. Even when such systems only “assist” investigators, poor accuracy can still subject vulnerable citizens to stressful scrutiny and administrative disruption. [The Guardian]theguardian.comOfficial figures, revealed through freedom of information laws, indicate that two-thirds of these flagged claims were legitimate. Consequ…

Human review illustration 3

What meaningful human judgement actually requires

Keeping humans “in the loop” only works if institutions actively preserve the conditions for judgement.

In practice, meaningful oversight usually requires several things at once.

Officials must understand the system

Reviewers need enough training to recognise uncertainty, failure modes, and bias risks. If the system appears mysterious or mathematically authoritative, staff may become reluctant to challenge it.

Humans need real authority to disagree

Officials must be able to override recommendations without punishment or excessive friction. Otherwise review becomes symbolic.

Appeals must be accessible

Citizens need practical routes to challenge decisions, especially when automated systems influenced the outcome.

Systems should expose uncertainty

Well-designed tools often display confidence levels, missing information, or reasons for recommendations rather than presenting outputs as objective truth.

Governments need audit trails

Agencies should be able to reconstruct how decisions were reached, who approved them, and what data influenced them.

These requirements may sound bureaucratic, but they are central to whether AI strengthens or weakens democratic governance. In many cases, the “boring” institutional layer matters more than the sophistication of the model itself.

The larger question behind human-centred public AI

The broader AI bloom vision imagines a future where intelligence becomes far more abundant and governments can coordinate resources, knowledge, and infrastructure far more effectively than today. In optimistic scenarios, advanced AI could help societies manage disasters better, reduce bureaucratic waste, accelerate scientific administration, improve healthcare access, and support more responsive public services.

But the public sector exposes a core tension inside that vision.

The more capable AI systems become, the stronger the temptation to automate governance itself. If systems appear more statistically accurate than individual officials, political pressure may grow to let them make increasingly consequential decisions directly.

Yet democratic societies are not organised solely around predictive accuracy. They are also organised around accountability, contestability, dignity, and the idea that citizens should not be governed entirely as data points.

That does not mean governments should reject advanced AI. It means the highest-value public uses may often be those that amplify human capacity rather than replace political and legal responsibility. The strongest systems may therefore be neither purely human nor fully automated, but institutions where AI expands what officials can understand while humans remain visibly answerable for the exercise of public power.

Endnotes

Source: arxiv.org
Title: arXiv Algorithms and Decision-Making in the Public Sector
Link: https://arxiv.org/abs/2106.03673
Source: arxiv.org
Link: https://arxiv.org/abs/2103.02381
Source snippet
arXivHuman-AI Interactions in Public Sector Decision-Making: "Automation Bias" and "Selective Adherence" to Algorithmic AdviceMarch 3, 2021...

Published: March 3, 2021
Source: academic.oup.com
Link: https://academic.oup.com/jpart/article/33/1/153/6524536
Source snippet
Artificial intelligence algorithms are increasingly adopted as decisional aides by public bodies, with the promise of overcoming biases o...
Source: arxiv.org
Link: https://arxiv.org/pdf/2105.01434
Source snippet
Towards Accountability in the Use of Artificial Intelligence...by M Loi · 2021 · Cited by 150 — We analyze the regulatory content of 16...
Source: ohchr.org
Title: landmark ruling dutch court stops government attempts spy poor un expert
Link: https://www.ohchr.org/en/press-releases/2020/02/landmark-ruling-dutch-court-stops-government-attempts-spy-poor-un-expert
Source snippet
OHCHRLandmark ruling by Dutch court stops government attempts...5 Feb 2020 — The court ordered the immediate halt to a digital benefit f...
Source: rechtspraak.nl
Link: https://www.rechtspraak.nl/organisatie-en-contact/organisatie/rechtbanken/rechtbank-den-haag/nieuws/syri-legislation-in-breach-of-european-convention-on-human-rights
Source snippet
RechtspraakSyRI legislation in breach of European Convention on...The Hague District Court has delivered a judgment today in a case abou...
Source: wired.com
Link: https://www.wired.com/story/gcse-results-alevels-algorithm-explained
Source snippet
The algorithm, designed by exams regulator Ofqual, aimed to standardize results nationally but ended up downgrading 40% of predicted grad...
Source: arxiv.org
Link: https://arxiv.org/abs/2604.22765
Source snippet
arXivAlgorithmic Administration and the EU AI Act: Legal Principles for Public Sector Use of AIMarch 24, 2026...

Published: March 24, 2026
Source: arxiv.org
Link: https://arxiv.org/pdf/2509.23843
Source snippet
(2020) Welfare surveillance system violates human rights, Dutch court rules.Read...
Source: arxiv.org
Link: https://arxiv.org/abs/2509.23843
Source snippet
Digital welfare fraud detection and the Dutch SyRI judgmentby M van Bekkum · 2025 · Cited by 168 — The court ruled that the SyRI legislat...
Source: opengovpartnership.org
Link: https://www.opengovpartnership.org/wp-content/uploads/2021/08/algorithmic-accountability-public-sector.pdf
Source snippet
Open Government PartnershipAlgorithmic accountability for the public sectorThis executive summary highlights the key findings from this s...
Source: loc.gov
Link: https://www.loc.gov/item/global-legal-monitor/2020-03-13/netherlands-court-prohibits-governments-use-of-ai-software-to-detect-welfare-fraud/
Source snippet
The Library of CongressNetherlands: Court Prohibits Government's Use of AI...The SyRI system is an algorithm used by the government to p...
Source: blogs.lse.ac.uk
Title: fk the algorithm what the world can learn from the uks a level grading fiasco
Link: https://blogs.lse.ac.uk/impactofsocialsciences/2020/08/26/fk-the-algorithm-what-the-world-can-learn-from-the-uks-a-level-grading-fiasco/
Source snippet
This is a well-established problem that data professionals have sought to...Read more...
Source: globalgovernmentforum.com
Title: uk government drops exam grading algorithm in the face of public anger
Link: https://www.globalgovernmentforum.com/uk-government-drops-exam-grading-algorithm-in-the-face-of-public-anger/
Source snippet
UK government drops exam grading algorithm in the face...18 Aug 2020 — The UK government has been forced to overturn exam results produc...
Source: theguardian.com
Link: https://www.theguardian.com/society/article/2024/jun/23/dwp-algorithm-wrongly-flags-200000-people-possible-fraud-error
Source snippet
Official figures, revealed through freedom of information laws, indicate that two-thirds of these flagged claims were legitimate. Consequ...
Source: theguardian.com
Link: https://www.theguardian.com/technology/2023/oct/23/uk-officials-use-ai-to-decide-on-issues-from-benefits-to-marriage-licences
Source snippet
This utilization ranges from automated passport gates to algorithms detecting fraudulent claims or sham marriages, with some results pote...
Source: dictionary.cambridge.org
Link: https://dictionary.cambridge.org/us/dictionary/english/algorithmic
Source snippet
| definition in the Cambridge English Dictionaryconnected with or using algorithms that control what someone is shown on a computer appli...
Source: Wikipedia
Link: https://en.wikipedia.org/wiki/Algorithmic
Source snippet
AlgorithmicAlgorithmic may refer to: Algorithm, step-by-step instructions for a calculation. Algorithmic art, art made by an algorithm...
Source: vocabulary.com
Link: https://www.vocabulary.com/dictionary/algorithmic
Source snippet
Definition, Meaning & Synonymsadjective of or relating to or having the characteristics of an algorithm synonyms: recursive of or relatin...
Source: theguardian.com
Title: welfare surveillance system violates human rights dutch court rules
Link: https://www.theguardian.com/technology/2020/feb/05/welfare-surveillance-system-violates-human-rights-dutch-court-rules
Source snippet
Welfare surveillance system violates human rights, Dutch...5 Feb 2020 — A Dutch court has ordered the immediate halt of an automated sur...

Additional References

Source: researchgate.net
Link: https://www.researchgate.net/publication/401421097_AI-ENABLED_PUBLIC_GOVERNANCE_IN_DEVELOPING_STATES_SERVICE_DELIVERY_GAINS_ACCOUNTABILITY_RISKS_AND_A_PRACTICAL_RISK-BASED_REGULATORY_MODEL
Source snippet
ai-enabled public governance in developing statesMar 4, 2026 — AI-ENABLED PUBLIC GOVERNANCE IN DEVELOPING STATES: SERVICE DELIVERY GAINS...
Source: collinsdictionary.com
Link: https://www.collinsdictionary.com/us/dictionary/english/algorithmic
Source snippet
ALGORITHMIC definition in American English1. a logical arithmetical or computational procedure that if correctly applied ensures the solu...
Source: sciencespo.fr
Link: https://www.sciencespo.fr/centre-etudes-europeennes/sites/sciencespo.fr.centre-etudes-europeennes/files/AccountableAI_PAR_Busuioc.pdf
Source snippet
Abstract: Artificial intelligence (AI) algorithms govern in subtle yet fundamental ways the way we live and are transforming our societie...
Source: naviant.com
Link: https://naviant.com/blog/human-in-the-loop-ai-government/
Source snippet
Human-in-the-Loop AI for the Public SectorHuman-in-the-loop means keeping people actively involved in AI-driven processes, building in hu...
Source: julia-project.eu
Link: https://www.julia-project.eu/sites/default/files/2025-05/Final%20Handbook_%20AI%20and%20Public%20Administration_%20The%20%28legal%29%20limits%20of%20algorithmic%20governance.docx.pdf
Source snippet
Handbook: AI and Public Administration: The (legal) limits...The JuLIA Handbook on AI and Public Administration: The (legal) limits of a...
Source: officialblogofunio.com
Link: https://officialblogofunio.com/2020/10/26/artificial-intelligence-2020-a-level-grades-in-the-uk-as-an-example-of-the-challenges-and-risks/
Source snippet
Artificial intelligence: 2020 A-level grades in the UK as an...26 Oct 2020 — This year A-Level students were graded by Ofqual, the natio...
Source: opengovpartnership.org
Link: https://www.opengovpartnership.org/wp-content/uploads/2021/08/executive-summary-algorithmic-accountability.pdf
Source: humanrightspulse.com
Title: dutch court finds syri algorithm violates human rights norms in landmark case
Link: https://www.humanrightspulse.com/mastercontentblog/dutch-court-finds-syri-algorithm-violates-human-rights-norms-in-landmark-case
Source snippet
Dutch court finds SyRI algorithm violates...Mar 22, 2020 — The court found SyRI was in violation of Article 8 (the right to respect for...
Source: oecd.org
Title: ai in public service design and delivery 09704c1a
Link: [https://www.oecd.org/en/publications/2025/06/governing
Source snippet
AI in public service design and delivery: Governing with...Sep 18, 2025 — The development and use of AI has permeated public service des...
Source: loti.london
Title: hil how can and should officers intervene in ai powered public services
Link: https://loti.london/blog/hil-how-can-and-should-officers-intervene-in-ai-powered-public-services/
Source snippet
Humans in the Loop: What should the role of officers be...Mar 4, 2025 — This role typically involves a service-level officer who evaluat...