AI Persuasion as a Threat to Human Control

• URL: http://arxiv.org/abs/2609.14796v1
• Date: Sun, 13 Sep 2026 21:17:57 GMT
• Title: AI Persuasion as a Threat to Human Control
• Abstract summary: We analyze how AI could persuade humans in key settings toward decisions that compromise the development, containment, oversight, and governance of AI itself.<n>We develop five concrete scenarios using this framework, and provide a blueprint for assessing the associated risks.<n>Our hope is that this paper highlights the risks from AI persuasion undermining control, and provides a path forward for future research.
• Score: 3.7466967375076003
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The threat that AI persuasion poses to human control has been acknowledged in the literature, but not yet systematically studied. Now that persuasion attacks are no longer theoretical - with Anthropic's Claude Mythos 5 recently making headlines for trying to convince people involved in an open-source project to merge malicious code during an evaluation - there is a pressing need to deeply analyze this threat. We undertake that effort here. In particular, we analyze how AI could persuade humans in key settings (e.g. safety-relevant R&D within frontier labs) toward decisions that compromise the development, containment, oversight, and governance of AI itself. In doing so, we elucidate a framework for characterizing this threat, develop five concrete scenarios using this framework, and provide a blueprint for assessing the associated risks. Using this blueprint, we conduct an initial risk estimation survey with select researchers and find that their opinions on which scenarios are riskiest are highly mixed. Their disagreements stem from differing opinions about the effectiveness of AI persuasion in different contexts, and point to the need for follow-up risk elicitation studies and persuasion evaluations, which we outline. Our hope is that this paper highlights the risks from AI persuasion undermining control, and provides a path forward for future research.

Related papers

• Frontier AI Risk Management Framework in Practice: A Risk Analysis Technical Report v1.5 [61.787178868669265]
  This technical report presents an updated and granular assessment of five critical dimensions: cyber offense, persuasion and manipulation, strategic deception, uncontrolled AI R&D, and self-replication.<n>This work reflects our current understanding of AI frontier risks and urges collective action to mitigate these challenges.
  arXiv  Detail & Related papers  (2026-02-16T04:30:06Z)
• Manipulation Attacks by Misaligned AI: Risk Analysis and Safety Case Framework [0.0]
  Humans are often the weakest link in cybersecurity systems.<n>A misaligned AI system may seek to undermine human oversight by manipulating employees.<n>No systematic framework exists for assessing and mitigating these risks.<n>This paper provides the first systematic methodology for integrating manipulation risk into AI safety governance.
  arXiv  Detail & Related papers  (2025-07-17T07:45:53Z)
• Persuasion and Safety in the Era of Generative AI [0.0]
  The EU AI Act prohibits AI systems that use manipulative or deceptive techniques to undermine informed decision-making.<n>My dissertation addresses the lack of empirical studies in this area by developing a taxonomy of persuasive techniques.<n>It provides resources to mitigate the risks of persuasive AI and fosters discussions on ethical persuasion in the age of generative AI.
  arXiv  Detail & Related papers  (2025-05-18T06:04:46Z)
• Must Read: A Systematic Survey of Computational Persuasion [60.83151988635103]
  AI-driven persuasion can be leveraged for beneficial applications, but also poses threats through manipulation and unethical influence.<n>Our survey outlines future research directions to enhance the safety, fairness, and effectiveness of AI-powered persuasion.
  arXiv  Detail & Related papers  (2025-05-12T17:26:31Z)
• Fully Autonomous AI Agents Should Not be Developed [50.61667544399082]
  This paper argues that fully autonomous AI agents should not be developed.<n>In support of this position, we build from prior scientific literature and current product marketing to delineate different AI agent levels.<n>Our analysis reveals that risks to people increase with the autonomy of a system.
  arXiv  Detail & Related papers  (2025-02-04T19:00:06Z)
• Particip-AI: A Democratic Surveying Framework for Anticipating Future AI Use Cases, Harms and Benefits [54.648819983899614]
  General purpose AI seems to have lowered the barriers for the public to use AI and harness its power. We introduce PARTICIP-AI, a framework for laypeople to speculate and assess AI use cases and their impacts.
  arXiv  Detail & Related papers  (2024-03-21T19:12:37Z)
• Risks of AI Scientists: Prioritizing Safeguarding Over Autonomy [65.77763092833348]
  This perspective examines vulnerabilities in AI scientists, shedding light on potential risks associated with their misuse.<n>We take into account user intent, the specific scientific domain, and their potential impact on the external environment.<n>We propose a triadic framework involving human regulation, agent alignment, and an understanding of environmental feedback.
  arXiv  Detail & Related papers  (2024-02-06T18:54:07Z)
• Control Risk for Potential Misuse of Artificial Intelligence in Science [85.91232985405554]
  We aim to raise awareness of the dangers of AI misuse in science. We highlight real-world examples of misuse in chemical science. We propose a system called SciGuard to control misuse risks for AI models in science.
  arXiv  Detail & Related papers  (2023-12-11T18:50:57Z)
• AI Risk Skepticism, A Comprehensive Survey [1.370633147306388]
  The study takes into account different points of view on the topic and draws parallels with other forms of skepticism that have shown up in science. We categorize the various skepticisms regarding the dangers of AI by the type of mistaken thinking involved.
  arXiv  Detail & Related papers  (2023-02-16T16:32:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.