AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathways

• URL: http://arxiv.org/abs/2406.02630v2
• Date: Fri, 6 Sep 2024 01:31:22 GMT
• Title: AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathways
• Authors: Zehang Deng, Yongjian Guo, Changzhou Han, Wanlun Ma, Junwu Xiong, Sheng Wen, Yang Xiang,
• Abstract summary: An Artificial Intelligence (AI) agent is a software entity that autonomously performs tasks or makes decisions based on pre-defined objectives and data inputs. This survey delves into the emerging security threats faced by AI agents, categorizing them into four critical knowledge gaps. By systematically reviewing these threats, this paper highlights both the progress made and the existing limitations in safeguarding AI agents.
• Score: 10.16690494897609
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: An Artificial Intelligence (AI) agent is a software entity that autonomously performs tasks or makes decisions based on pre-defined objectives and data inputs. AI agents, capable of perceiving user inputs, reasoning and planning tasks, and executing actions, have seen remarkable advancements in algorithm development and task performance. However, the security challenges they pose remain under-explored and unresolved. This survey delves into the emerging security threats faced by AI agents, categorizing them into four critical knowledge gaps: unpredictability of multi-step user inputs, complexity in internal executions, variability of operational environments, and interactions with untrusted external entities. By systematically reviewing these threats, this paper highlights both the progress made and the existing limitations in safeguarding AI agents. The insights provided aim to inspire further research into addressing the security threats associated with AI agents, thereby fostering the development of more robust and secure AI agent applications.

Related papers

• Multi-Agent Risks from Advanced AI [90.74347101431474]
  Multi-agent systems of advanced AI pose novel and under-explored risks. We identify three key failure modes based on agents' incentives, as well as seven key risk factors. We highlight several important instances of each risk, as well as promising directions to help mitigate them.
  arXiv  Detail & Related papers  (2025-02-19T23:03:21Z)
• Towards Robust and Secure Embodied AI: A Survey on Vulnerabilities and Attacks [22.154001025679896]
  Embodied AI systems, including robots and autonomous vehicles, are increasingly integrated into real-world applications. These vulnerabilities manifest through sensor spoofing, adversarial attacks, and failures in task and motion planning.
  arXiv  Detail & Related papers  (2025-02-18T03:38:07Z)
• Attack Atlas: A Practitioner's Perspective on Challenges and Pitfalls in Red Teaming GenAI [52.138044013005]
  generative AI, particularly large language models (LLMs), become increasingly integrated into production applications. New attack surfaces and vulnerabilities emerge and put a focus on adversarial threats in natural language and multi-modal systems. Red-teaming has gained importance in proactively identifying weaknesses in these systems, while blue-teaming works to protect against such adversarial attacks. This work aims to bridge the gap between academic insights and practical security measures for the protection of generative AI systems.
  arXiv  Detail & Related papers  (2024-09-23T10:18:10Z)
• Safeguarding AI Agents: Developing and Analyzing Safety Architectures [0.0]
  This paper addresses the need for safety measures in AI systems that collaborate with human teams. We propose and evaluate three frameworks to enhance safety protocols in AI agent systems. We conclude that these frameworks can significantly strengthen the safety and security of AI agent systems.
  arXiv  Detail & Related papers  (2024-09-03T10:14:51Z)
• EARBench: Towards Evaluating Physical Risk Awareness for Task Planning of Foundation Model-based Embodied AI Agents [53.717918131568936]
  Embodied artificial intelligence (EAI) integrates advanced AI models into physical entities for real-world interaction. Foundation models as the "brain" of EAI agents for high-level task planning have shown promising results. However, the deployment of these agents in physical environments presents significant safety challenges. This study introduces EARBench, a novel framework for automated physical risk assessment in EAI scenarios.
  arXiv  Detail & Related papers  (2024-08-08T13:19:37Z)
• Security of AI Agents [5.468745160706382]
  We identify and describe potential vulnerabilities in AI agents in detail from a system security perspective. We introduce defense mechanisms corresponding to each vulnerability with design and experiments to evaluate their viability. This paper contextualizes the security issues in the current development of AI agents and delineates methods to make AI agents safer and more reliable.
  arXiv  Detail & Related papers  (2024-06-12T23:16:45Z)
• Artificial Intelligence as the New Hacker: Developing Agents for Offensive Security [0.0]
  This paper explores the integration of Artificial Intelligence (AI) into offensive cybersecurity. It develops an autonomous AI agent, ReaperAI, designed to simulate and execute cyberattacks. ReaperAI demonstrates the potential to identify, exploit, and analyze security vulnerabilities autonomously.
  arXiv  Detail & Related papers  (2024-05-09T18:15:12Z)
• Testing autonomous vehicles and AI: perspectives and challenges from cybersecurity, transparency, robustness and fairness [53.91018508439669]
  The study explores the complexities of integrating Artificial Intelligence into Autonomous Vehicles (AVs) It examines the challenges introduced by AI components and the impact on testing procedures. The paper identifies significant challenges and suggests future directions for research and development of AI in AV technology.
  arXiv  Detail & Related papers  (2024-02-21T08:29:42Z)
• Managing extreme AI risks amid rapid progress [171.05448842016125]
  We describe risks that include large-scale social harms, malicious uses, and irreversible loss of human control over autonomous AI systems. There is a lack of consensus about how exactly such risks arise, and how to manage them. Present governance initiatives lack the mechanisms and institutions to prevent misuse and recklessness, and barely address autonomous systems.
  arXiv  Detail & Related papers  (2023-10-26T17:59:06Z)
• AI Potentiality and Awareness: A Position Paper from the Perspective of Human-AI Teaming in Cybersecurity [18.324118502535775]
  We argue that human-AI teaming is worthwhile in cybersecurity. We emphasize the importance of a balanced approach that incorporates AI's computational power with human expertise.
  arXiv  Detail & Related papers  (2023-09-28T01:20:44Z)
• Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims [59.64274607533249]
  AI developers need to make verifiable claims to which they can be held accountable. This report suggests various steps that different stakeholders can take to improve the verifiability of claims made about AI systems. We analyze ten mechanisms for this purpose--spanning institutions, software, and hardware--and make recommendations aimed at implementing, exploring, or improving those mechanisms.
  arXiv  Detail & Related papers  (2020-04-15T17:15:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.