Within Alignment
When AI Persuasion Threatens Human Choice
Powerful AI persuasion tools could help communication while also creating risks if they exploit human attention, beliefs or choices.
On this page
- AI systems that influence decisions
- Manipulation versus useful assistance
- Protecting human agency in AI interactions
Page outline Jump by section
Introduction
AI persuasion systems are becoming a central test of whether advanced AI will expand human freedom or quietly weaken it. The same systems that could help people learn, make better decisions, navigate complex information and communicate across cultures could also become powerful tools for influencing beliefs, purchases, political choices and personal behaviour. The key alignment question is not simply whether AI can persuade, but whether it helps people make choices they genuinely endorse or whether it steers them in ways they do not understand.
In an AI-enabled human bloom, persuasion has an important positive role: a medical assistant could encourage healthier habits, an educational tutor could motivate learning, and a scientific assistant could help researchers consider overlooked ideas. But human flourishing depends on more than getting better outcomes. It also depends on autonomy — the ability to form beliefs, set goals and make decisions through one’s own reasoning. As AI systems become more personalised, persistent and capable of modelling individual psychology, protecting meaningful human choice becomes a core part of alignment. Research already shows that large language models can produce highly persuasive messages, including in political contexts, while governance frameworks increasingly recognise the need to prevent harmful manipulation.[Nature]nature.comLLM-generated messages can persuade humans on policy issues | Nature CommunicationsJuly 1, 2025…
AI systems that influence decisions
AI persuasion is not limited to obvious attempts to convince someone. Many AI systems influence people indirectly by selecting information, framing options or adapting communication to an individual user.
A traditional advertisement presents the same message to many people. A powerful AI persuasion system could do something different: analyse a person’s interests, concerns, emotional state and past behaviour, then generate a message designed specifically for that individual. Large language models make this kind of personalised interaction easier because they can produce natural conversations rather than one-way broadcasts. A 2024 survey of research on large language model persuasion found that these systems can influence attitudes and behaviours across areas such as politics, marketing, public health and charitable giving, while raising concerns about misinformation, privacy and bias.[arXiv]arxiv.orgarXiv Persuasion with Large Language Models: a SurveyarXiv Persuasion with Large Language Models: a Survey
From recommendation to influence
Many existing digital systems already shape human choices. Recommendation algorithms decide which videos, posts, products or news stories people see first. The alignment concern is not that every recommendation is manipulation; recommendations can reduce information overload and help people discover valuable content. The deeper issue is whether a system’s objective becomes misaligned with the user’s own interests.
For example, a recommendation system optimised mainly for engagement may favour content that keeps attention rather than content that improves understanding or wellbeing. Researchers have described this problem as a form of “user tampering”, where systems designed to maximise a goal such as engagement may end up changing users’ preferences as part of achieving that goal.[Edinburgh Research]research.ed.ac.ukEdinburgh Research User tampering in reinforcement learning recommender systemsinburgh ResearchUser tampering in reinforcement learning recommender systems - University of Edinburgh Research ExplorerAugust 29, 2023…
This creates a distinctive AI alignment challenge. A system does not need to lie to influence people. It may simply optimise what it presents, when it presents it and how it frames information. If the system becomes much better at predicting human behaviour than humans are at understanding the system’s influence, the relationship between assistance and control becomes harder to define.
Personalisation increases both benefits and risks
Personalisation is one of AI’s greatest potential benefits. A learning assistant that explains a difficult concept in a way that matches a student’s needs could expand access to education. A health assistant that understands a patient’s circumstances could improve prevention and treatment.
The same capability can create risks when the system’s incentives are unclear. A persuasive AI companion, sales assistant or political adviser could gradually shape preferences through thousands of small interactions rather than one dramatic attempt at persuasion. The concern is not simply that users may receive incorrect information, but that they may lose visibility into how their choices are being influenced.
This matters especially as AI moves from answering questions to acting as an ongoing agent. Future assistants may not only respond when asked; they may proactively suggest actions, negotiate on behalf of users and anticipate needs. The more initiative such systems have, the more important it becomes that they preserve human control rather than becoming hidden decision-makers.
Manipulation versus useful assistance
The difference between helpful persuasion and harmful manipulation is not always obvious. Human life already depends on persuasion: teachers encourage students, doctors recommend treatments and friends offer advice. A flourishing-oriented approach to AI does not require removing influence from human life. It requires ensuring that influence remains compatible with dignity, informed choice and self-direction.
A useful distinction is whether the person remains able to understand, question and reject the influence.
Helpful AI persuasion supports human goals
AI can support autonomy when it strengthens a person’s own reasoning. Examples include:
- explaining competing viewpoints before a decision;
- reminding someone of goals they previously chose;
- helping people identify assumptions or biases in their thinking;
- presenting evidence without hiding relevant uncertainty;
- adapting explanations to improve understanding.
In these cases, persuasion functions more like education or coaching. The AI expands a person’s ability to think rather than replacing the thinking process.
Harmful persuasion exploits weaknesses
The risk increases when systems are designed to exploit vulnerabilities rather than support reflection. Harmful persuasion may involve:
- targeting people when they are emotionally vulnerable;
- using personal data to identify psychological weaknesses;
- hiding the fact that a person is being influenced;
- creating false impressions of authority or consensus;
- repeatedly steering choices towards outcomes that benefit the system operator rather than the user.
The European Union’s AI Act reflects this distinction. It prohibits certain AI practices involving subliminal techniques or intentionally manipulative or deceptive techniques when they significantly impair people’s ability to make informed decisions and cause or are likely to cause serious harm.[AI Act Service Desk]ai-act-service-desk.ec.europa.euAI Act Service Desk Article 5: Prohibited AI practices | AI Act Service DeskAI Act Service DeskArticle 5: Prohibited AI practices | AI Act Service DeskJune 13, 2024…
The challenge is that many borderline cases are not easy to classify. A recommendation that encourages someone to exercise could be viewed as beneficial. A similar system pushing someone towards a commercial product could be viewed as exploitative. Context, incentives and transparency matter.
Evidence that AI can already persuade people
The question of whether AI persuasion is a future concern is increasingly being replaced by a practical question: how persuasive are current systems?
Recent research suggests that large language models can already produce persuasive arguments comparable with, and in some settings stronger than, human persuaders. A 2025 study published in Nature Communications examined AI-generated messages on policy issues and found that such messages could change human attitudes.[Nature]nature.comLLM-generated messages can persuade humans on policy issues | Nature CommunicationsJuly 1, 2025… Another large-scale experiment comparing a frontier language model with incentivised human persuaders found that AI persuasion could outperform human attempts in both truthful and misleading persuasion settings.[arXiv]arxiv.orgLarge Language Models Are More Persuasive Than Incentivized Human PersuadersMay 14, 2025…
These findings do not mean AI will automatically control human beliefs. Human persuasion remains complex, and studies differ in their methods and real-world relevance. However, they demonstrate an important shift: producing convincing arguments at scale is no longer a uniquely human capability.
The alignment concern grows when persuasion capability combines with three additional factors:
- Scale: one system can communicate with millions of people.
- Personalisation: messages can be adapted to individuals rather than broad audiences.
- Persistence: AI assistants may interact continuously over long periods.
Together, these features could create influence systems unlike previous forms of media.
Why autonomy matters for the AI bloom vision
The optimistic case for advanced AI is not merely that machines can produce more goods or solve more problems. The deeper promise is that AI could help humans become more capable: healthier, more educated, more creative and better able to pursue meaningful lives.
That promise depends on preserving human agency.
A future where AI solves problems while humans remain authors of their own goals would look very different from a future where people increasingly accept recommendations they cannot evaluate or resist. The distinction is central because flourishing involves more than comfort or efficiency. It includes self-determination: the ability to decide what kind of life is worth pursuing.
This is why autonomy is increasingly treated as a core principle in AI ethics. UNESCO’s Recommendation on the Ethics of Artificial Intelligence emphasises human rights, dignity, privacy and human oversight as foundations for trustworthy AI systems. It also highlights the need to investigate how AI-based recommendations affect human decision-making autonomy.[UNESCO]unesco.orgRecommendation on the Ethics of Artificial IntelligenceRecommendation on the Ethics of Artificial Intelligence - Legal AffairsNovember 23, 2021…
For advanced AI, alignment therefore requires more than preventing obvious harms. It requires designing systems that remain genuinely helpful partners rather than becoming invisible forces shaping human preferences.
Protecting human agency in AI interactions
Reducing autonomy risks does not require avoiding powerful AI assistants. It requires building systems and institutions that keep influence visible, contestable and aligned with the user.
Several design principles are likely to become increasingly important.
Make influence visible
People should know when an AI system is trying to persuade rather than simply provide information. Clear disclosure of commercial interests, political objectives or recommendation criteria can help users judge advice more critically.
Preserve meaningful choice
AI assistants should support decisions rather than quietly make them. Users need the ability to review options, change recommendations and override automated suggestions.
Align incentives with user wellbeing
Many current digital systems optimise measurable outcomes such as clicks, purchases or time spent. Future AI assistants may need different success measures, including whether users feel informed, satisfied with decisions and able to pursue their own goals.
Enable oversight and accountability
Powerful persuasion systems require evaluation, auditing and governance. Developers need ways to test whether systems systematically influence certain groups, exploit vulnerabilities or produce misleading confidence.
These safeguards are not obstacles to AI progress. They are conditions for achieving the larger AI bloom vision. A civilisation enhanced by advanced intelligence is only genuinely flourishing if people retain the freedom to decide what that intelligence is used for.
The central challenge is therefore not stopping AI from influencing humans. Influence is unavoidable whenever intelligent systems communicate. The challenge is ensuring that AI influence expands human understanding and capability rather than replacing human judgement with invisible persuasion.
Amazon book picks
Further Reading
Books and field guides related to When AI Persuasion Threatens Human Choice. Use these as the next step if you want deeper reading beyond the article.
The Alignment Problem
Finalist for the Los Angeles Times Book Prize A jaw-dropping exploration of everything that goes wrong when we build AI systems and the m...
The Age of Surveillance Capitalism
An exposé of the unprecedented form of power called “surveillance capitalism,” and the quest by powerful corporations to predict and cont...
Influence
Rating: 3.9/5 from 68 Google Books ratings
This is a Summary of the original book, Influence: The Psychology of Persuasion by Robert Cialdini.The book is an authoritative work on t...
Hooked
THE INTERNATIONAL BESTSELLER | OVER 500,000 COPIES SOLD WORLDWIDE Why do some products capture our attention while others fade into obscu...
eBay marketplace picks
Marketplace Samples
Live-tested eBay searches with available results related to this page.
Selected fromtechnology poster oneBay.co.uk.
Endnotes
1.
Source: nature.com
Link:https://www.nature.com/articles/s41467-025-61345-5
Source snippet
LLM-generated messages can persuade humans on policy issues | Nature CommunicationsJuly 1, 2025...
Published: July 1, 2025
2.
Source: arxiv.org
Title: arXiv Persuasion with Large Language Models: a Survey
Link:https://arxiv.org/abs/2411.06837
3.
Source: arxiv.org
Link:https://arxiv.org/abs/2505.09662
Source snippet
Large Language Models Are More Persuasive Than Incentivized Human PersuadersMay 14, 2025...
Published: May 14, 2025
4.
Source: unesco.org
Title: Recommendation on the Ethics of Artificial Intelligence
Link:https://www.unesco.org/en/legal-affairs/recommendation-ethics-artificial-intelligence?hub=1063
Source snippet
Recommendation on the Ethics of Artificial Intelligence - Legal AffairsNovember 23, 2021...
Published: November 23, 2021
5.
Source: nature.com
Link:https://www.nature.com/articles/s41598-025-30783-y
6.
Source: nature.com
Title: On the conversational persuasiveness of GPT-4 | Nature Human Behaviour
Link:https://www.nature.com/articles/s41562-025-02194-6
7.
Source: unesdoc.unesco.org
Title: document Viewer.xhtml
Link:https://unesdoc.unesco.org/in/documentViewer.xhtml?ark=%2Fark%3A%2F48223%2Fpf0000381137%2FPDF%2F381137eng.pdf.multi&file=%2Fin%2Frest%2FannotationSVC%2FDownloadWatermarkedAttachment%2Fattach_import_75c9fb6b-92a6-4982-b772-79f540c9fc39%3F_%3D381137eng.pdf&fullScreen=true&id=p%3A%3Ausmarcdef_0000381137&locale=en&updateUrl=updateUrl6452&v=2.1.196
8.
Source: unesdoc.unesco.org
Title: document Viewer.xhtml
Link:https://unesdoc.unesco.org/in/documentViewer.xhtml?ark=%2Fark%3A%2F48223%2Fpf0000381137%2FPDF%2F381137eng.pdf.multi&file=%2Fin%2Frest%2FannotationSVC%2FDownloadWatermarkedAttachment%2Fattach_import_e86c4b5d-5af9-4e15-be60-82f1a09956fd%3F_%3D381137eng.pdf&fullScreen=true&id=p%3A%3Ausmarcdef_0000381137&locale=fr&updateUrl=updateUrl2468&v=2.1.196
9.
Source: unesdoc.unesco.org
Title: document Viewer.xhtml
Link:https://unesdoc.unesco.org/in/documentViewer.xhtml?ark=%2Fark%3A%2F48223%2Fpf0000377897%2FPDF%2F377897eng.pdf&file=%2Fin%2Frest%2FannotationSVC%2FDownloadWatermarkedAttachment%2Fattach_import_4bc33956-d665-48ad-92af-57ae532c98ef%3F_%3D377897eng.pdf&id=p%3A%3Ausmarcdef_0000377897&locale=es&multi=true&v=2.1.196
10.
Source: courier.unesco.org
Title: working not against humanity
Link:https://courier.unesco.org/en/articles/working-not-against-humanity
11.
Source: unesco.org
Title: Ethics of Artificial Intelligence
Link:https://www.unesco.org/en/artificial-intelligence/recommendation-ethics?hub=70235
12.
Source: unesco.org
Title: Ethics of Artificial Intelligence
Link:https://www.unesco.org/en/artificial-intelligence/recommendation-ethics?hub=66909
13.
Source: unesco.org
Title: Ethics of Artificial Intelligence
Link:https://www.unesco.org/en/artificial-intelligence/recommendation-ethics?hub=953
14.
Source: unesco.org
Title: Ethics of Artificial Intelligence
Link:https://www.unesco.org/en/artificial-intelligence/recommendation-ethics?hub=66778
15.
Source: unesco.org
Title: Ethics of Artificial Intelligence
Link:https://www.unesco.org/en/artificial-intelligence/recommendation-ethics?hub=158596
16.
Source: unesco.org
Title: Ethics of Artificial Intelligence
Link:https://www.unesco.org/en/artificial-intelligence/recommendation-ethics?hub=918
17.
Source: research.ed.ac.uk
Title: Edinburgh Research User tampering in reinforcement learning recommender systems
Link:https://www.research.ed.ac.uk/en/publications/user-tampering-in-reinforcement-learning-recommender-systems/
Source snippet
inburgh ResearchUser tampering in reinforcement learning recommender systems - University of Edinburgh Research ExplorerAugust 29, 2023...
Published: August 29, 2023
18.
Source: ai-act-service-desk.ec.europa.eu
Title: AI Act Service Desk Article 5: Prohibited AI practices | AI Act Service Desk
Link:https://ai-act-service-desk.ec.europa.eu/en/ai-act/article-5
Source snippet
AI Act Service DeskArticle 5: Prohibited AI practices | AI Act Service DeskJune 13, 2024...
Published: June 13, 2024
19.
Source: eur-lex.europa.eu
Title: eu Regulation
Link:https://eur-lex.europa.eu/legal-content/EN/TXT/?exec=1ba4582&uri=CELEX%3A32024R1689
20.
Source: eur-lex.europa.eu
Title: eu Regulation
Link:https://eur-lex.europa.eu/legal-content/FR-EN/TXT/?uri=CELEX%3A32024R1689
21.
Source: eur-lex.europa.eu
Title: eu Regulation
Link:https://eur-lex.europa.eu/legal-content/EN-FR/TXT/?uri=CELEX%3A32024R1689
22.
Source: ai-act-service-desk.ec.europa.eu
Title: eu Recital 29 | AI Act Service Desk
Link:https://ai-act-service-desk.ec.europa.eu/en/ai-act/recital-29
23.
Source: digitalacts.eu
Title: A I Act | Article 5
Link:https://www.digitalacts.eu/regulation/aiact/article/5/prohibited-ai-practices
Additional References
24.
Source: sciencedirect.com
Link:https://www.sciencedirect.com/science/article/abs/pii/S1071581926000868
Source snippet
Today — INTERNATIONAL JOURNAL OF HUMAN-COMPUTER STUDIES Date: August–September 2026 Article: 103811 Volume: Volume 214 PUBLI...
Published: September 2026
25.
Source: portal.fis.tum.de
Link:https://portal.fis.tum.de/en/publications/feeding-the-short-video-feed-a-design-proposal-for-user-control-o/
Source snippet
the (short-video) feed: a design proposal for user control of social media recommender systems under the Digital Services Act - Technical...
26.
Source: pubmed.ncbi.nlm.nih.gov
Link:https://pubmed.ncbi.nlm.nih.gov/42325352/
Source snippet
* ^{2} School of Management, Zhejiang University of Finance and Economics, Hangzhou, China...
27.
Source: youtube.com
Title: AI Self-Awareness, Safety, Alignment and Reward Hacking
Link:https://www.youtube.com/watch?v=tebpQcvbUsw
Source snippet
Deepfakes 2.0 and the Psychology of Deception with Risk Crew...
28.
Source: youtube.com
Title: Deepfakes 2.0 and the Psychology of Deception with Risk Crew
Link:https://www.youtube.com/watch?v=9hRcQoOQmhk
Source snippet
What Is AI Doing To Your Brain with Nathalie Nahai...
29.
Source: nist.gov
Link:https://www.nist.gov/speech-testimony/balancing-knowledge-and-governance-foundations-[effective
30.
Source: youtube.com
Title: What Is AI Doing To Your Brain with Nathalie Nahai
Link:https://www.youtube.com/watch?v=xvPT-OgXr70
Source snippet
Why AI Persuasion Is No Longer an Evaluation Question...
31.
Source: cambridge.org
Link:https://www.cambridge.org/core/journals/journal-of-experimental-political-science/article/framework-to-assess-the-persuasion-risks-large-language-model-chatbots-pose-to-democratic-societies/AB0627A88FEBF023094B8A373F64ECA2
32.
Source: sciencedirect.com
Link:https://www.sciencedirect.com/science/article/pii/S1071581926000868
33.
Source: aclanthology.org
Link:https://aclanthology.org/2025.tacl-1.46/