Understanding Cognition-Induced Risks in Agentic AI Systems

• URL: http://arxiv.org/abs/2608.15304v1
• Date: Sat, 15 Aug 2026 16:16:29 GMT
• Title: Understanding Cognition-Induced Risks in Agentic AI Systems
• Abstract summary: Agentic systems powered by large language models (LLMs) exhibit human-like patterns of cognition.<n>As these systems become deeply integrated across different domains, their cognitive engagement raises critical concerns for human society that remain insufficiently studied.<n>We study their potential risks to human agency, autonomy, and control capability, corresponding to each cognitive level.
• Score: 46.75121501746157
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Frontier agentic systems powered by large language models (LLMs) exhibit human-like patterns of cognition. As these systems become deeply integrated across different domains, their cognitive engagement raises critical concerns for human society that remain insufficiently studied. To address this gap, we systematically analyze risks induced by expanding cognitive capabilities, following a three-level framework defined by their cognitive scope, from physical cognition to social cognition, and finally to self-referential cognition. We study their potential risks to human agency, autonomy, and control capability, corresponding to each cognitive level. We finally propose strategies to mitigate these risks and enhance the controllability of agentic AI systems, ensuring their long-term safe development.

Related papers

• An approach to systemic risks of AI through the lens of emergence, collective action problems, and externalities [41.99844472131922]
  There is no generally accepted definition of systemic risks in general and for AI in particular.<n>The application of the systemic risk approach to human rights or fundamental rights, like in the EU AI Act, is relatively new.<n>We argue that some concepts so far have not sufficiently take complexity and emergence into account.
  arXiv  Detail & Related papers  (2026-07-20T17:12:04Z)
• A Survey of Reasoning in Autonomous Driving Systems: Open Challenges and Emerging Paradigms [49.66022971508878]
  We argue that reasoning should be elevated from a modular component to the system's cognitive core.<n>We derive and systematize seven core reasoning challenges, such as the responsiveness-reasoning trade-off and social-game reasoning.<n>We identify a fundamental and unresolved tension between the high-latency, deliberative nature of LLM-based reasoning and the millisecond-scale, safety-critical demands of vehicle control.
  arXiv  Detail & Related papers  (2026-03-11T07:40:53Z)
• AI Deception: Risks, Dynamics, and Controls [153.71048309527225]
  This project provides a comprehensive and up-to-date overview of the AI deception field.<n>We identify a formal definition of AI deception, grounded in signaling theory from studies of animal deception.<n>We organize the landscape of AI deception research as a deception cycle, consisting of two key components: deception emergence and deception treatment.
  arXiv  Detail & Related papers  (2025-11-27T16:56:04Z)
• Identifying, Evaluating, and Mitigating Risks of AI Thought Partnerships [35.135633927249735]
  We identify risks of AI thought partners through a novel framework.<n>We propose concrete metrics for risk evaluation.<n>We suggest specific mitigation strategies for developers and policymakers.
  arXiv  Detail & Related papers  (2025-05-22T16:58:48Z)
• From Consumption to Collaboration: Measuring Interaction Patterns to Augment Human Cognition in Open-Ended Tasks [2.048226951354646]
  The rise of Generative AI, and Large Language Models (LLMs) in particular, is fundamentally changing cognitive processes in knowledge work.<n>We present a framework that analyzes interaction patterns along two dimensions: cognitive activity mode (exploration vs. exploitation) and cognitive engagement mode (constructive vs. detrimental)
  arXiv  Detail & Related papers  (2025-04-03T17:20:36Z)
• Agentic AI Needs a Systems Theory [46.36636351388794]
  We argue that AI development is currently overly focused on individual model capabilities.<n>We outline mechanisms for enhanced agent cognition, emergent causal reasoning ability, and metacognitive awareness.<n>We emphasize that a systems-level perspective is essential for better understanding, and purposefully shaping, agentic AI systems.
  arXiv  Detail & Related papers  (2025-02-28T22:51:32Z)
• Imagining and building wise machines: The centrality of AI metacognition [78.76893632793497]
  We examine what is known about human wisdom and sketch a vision of its AI counterpart.<n>We argue that AI systems particularly struggle with metacognition.<n>We discuss how wise AI might be benchmarked, trained, and implemented.
  arXiv  Detail & Related papers  (2024-11-04T18:10:10Z)
• Evaluating AI Evaluation: Perils and Prospects [8.086002368038658]
  This paper contends that the prevalent evaluation methods for these systems are fundamentally inadequate. I argue that a reformation is required in the way we evaluate AI systems and that we should look towards cognitive sciences for inspiration.
  arXiv  Detail & Related papers  (2024-07-12T12:37:13Z)
• Risks of AI Scientists: Prioritizing Safeguarding Over Autonomy [65.77763092833348]
  This perspective examines vulnerabilities in AI scientists, shedding light on potential risks associated with their misuse.<n>We take into account user intent, the specific scientific domain, and their potential impact on the external environment.<n>We propose a triadic framework involving human regulation, agent alignment, and an understanding of environmental feedback.
  arXiv  Detail & Related papers  (2024-02-06T18:54:07Z)
• PsySafe: A Comprehensive Framework for Psychological-based Attack, Defense, and Evaluation of Multi-agent System Safety [70.84902425123406]
  Multi-agent systems, when enhanced with Large Language Models (LLMs), exhibit profound capabilities in collective intelligence. However, the potential misuse of this intelligence for malicious purposes presents significant risks. We propose a framework (PsySafe) grounded in agent psychology, focusing on identifying how dark personality traits in agents can lead to risky behaviors. Our experiments reveal several intriguing phenomena, such as the collective dangerous behaviors among agents, agents' self-reflection when engaging in dangerous behavior, and the correlation between agents' psychological assessments and dangerous behaviors.
  arXiv  Detail & Related papers  (2024-01-22T12:11:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.