AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathways

• URL: http://arxiv.org/abs/2406.02630v1
• Date: Tue, 4 Jun 2024 01:22:31 GMT
• Title: AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathways
• Authors: Zehang Deng, Yongjian Guo, Changzhou Han, Wanlun Ma, Junwu Xiong, Sheng Wen, Yang Xiang,
• Abstract summary: An Artificial Intelligence (AI) agent is a software entity that autonomously performs tasks or makes decisions based on pre-defined objectives and data inputs. This survey delves into the emerging security threats faced by AI agents, categorizing them into four critical knowledge gaps. By systematically reviewing these threats, this paper highlights both the progress made and the existing limitations in safeguarding AI agents.
• Score: 10.16690494897609
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: An Artificial Intelligence (AI) agent is a software entity that autonomously performs tasks or makes decisions based on pre-defined objectives and data inputs. AI agents, capable of perceiving user inputs, reasoning and planning tasks, and executing actions, have seen remarkable advancements in algorithm development and task performance. However, the security challenges they pose remain under-explored and unresolved. This survey delves into the emerging security threats faced by AI agents, categorizing them into four critical knowledge gaps: unpredictability of multi-step user inputs, complexity in internal executions, variability of operational environments, and interactions with untrusted external entities. By systematically reviewing these threats, this paper highlights both the progress made and the existing limitations in safeguarding AI agents. The insights provided aim to inspire further research into addressing the security threats associated with AI agents, thereby fostering the development of more robust and secure AI agent applications.

Related papers

• Security of AI Agents [5.468745160706382]
  The study and development of AI agents have been boosted by large language models. In this paper, we identify and describe these vulnerabilities in detail from a system security perspective. We introduce defense mechanisms corresponding to each vulnerability with meticulous design and experiments to evaluate their viability.
  arXiv  Detail & Related papers  (2024-06-12T23:16:45Z)
• Artificial Intelligence as the New Hacker: Developing Agents for Offensive Security [0.0]
  This paper explores the integration of Artificial Intelligence (AI) into offensive cybersecurity. It develops an autonomous AI agent, ReaperAI, designed to simulate and execute cyberattacks. ReaperAI demonstrates the potential to identify, exploit, and analyze security vulnerabilities autonomously.
  arXiv  Detail & Related papers  (2024-05-09T18:15:12Z)
• Position Paper: Agent AI Towards a Holistic Intelligence [53.35971598180146]
  We emphasize developing Agent AI -- an embodied system that integrates large foundation models into agent actions. In this paper, we propose a novel large action model to achieve embodied intelligent behavior, the Agent Foundation Model.
  arXiv  Detail & Related papers  (2024-02-28T16:09:56Z)
• Testing autonomous vehicles and AI: perspectives and challenges from cybersecurity, transparency, robustness and fairness [53.91018508439669]
  The study explores the complexities of integrating Artificial Intelligence into Autonomous Vehicles (AVs) It examines the challenges introduced by AI components and the impact on testing procedures. The paper identifies significant challenges and suggests future directions for research and development of AI in AV technology.
  arXiv  Detail & Related papers  (2024-02-21T08:29:42Z)
• Managing extreme AI risks amid rapid progress [171.05448842016125]
  We describe risks that include large-scale social harms, malicious uses, and irreversible loss of human control over autonomous AI systems. There is a lack of consensus about how exactly such risks arise, and how to manage them. Present governance initiatives lack the mechanisms and institutions to prevent misuse and recklessness, and barely address autonomous systems.
  arXiv  Detail & Related papers  (2023-10-26T17:59:06Z)
• AI Potentiality and Awareness: A Position Paper from the Perspective of Human-AI Teaming in Cybersecurity [18.324118502535775]
  We argue that human-AI teaming is worthwhile in cybersecurity. We emphasize the importance of a balanced approach that incorporates AI's computational power with human expertise.
  arXiv  Detail & Related papers  (2023-09-28T01:20:44Z)
• AI for IT Operations (AIOps) on Cloud Platforms: Reviews, Opportunities and Challenges [60.56413461109281]
  Artificial Intelligence for IT operations (AIOps) aims to combine the power of AI with the big data generated by IT Operations processes. We discuss in depth the key types of data emitted by IT Operations activities, the scale and challenges in analyzing them, and where they can be helpful. We categorize the key AIOps tasks as - incident detection, failure prediction, root cause analysis and automated actions.
  arXiv  Detail & Related papers  (2023-04-10T15:38:12Z)
• Proceedings of the Artificial Intelligence for Cyber Security (AICS) Workshop at AAAI 2022 [55.573187938617636]
  The workshop will focus on the application of AI to problems in cyber security. Cyber systems generate large volumes of data, utilizing this effectively is beyond human capabilities.
  arXiv  Detail & Related papers  (2022-02-28T18:27:41Z)
• Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims [59.64274607533249]
  AI developers need to make verifiable claims to which they can be held accountable. This report suggests various steps that different stakeholders can take to improve the verifiability of claims made about AI systems. We analyze ten mechanisms for this purpose--spanning institutions, software, and hardware--and make recommendations aimed at implementing, exploring, or improving those mechanisms.
  arXiv  Detail & Related papers  (2020-04-15T17:15:35Z)
• Vulnerabilities of Connectionist AI Applications: Evaluation and Defence [0.0]
  This article deals with the IT security of connectionist artificial intelligence (AI) applications, focusing on threats to integrity. A comprehensive list of threats and possible mitigations is presented by reviewing the state-of-the-art literature. The discussion of mitigations is likewise not restricted to the level of the AI system itself but rather advocates viewing AI systems in the context of their supply chains.
  arXiv  Detail & Related papers  (2020-03-18T12:33:59Z)
• TanksWorld: A Multi-Agent Environment for AI Safety Research [5.218815947097599]
  The ability to create artificial intelligence capable of performing complex tasks is rapidly outpacing our ability to ensure the safe and assured operation of AI-enabled systems. Recent simulation environments to illustrate AI safety risks are relatively simple or narrowly-focused on a particular issue. We introduce the AI safety TanksWorld as an environment for AI safety research with three essential aspects.
  arXiv  Detail & Related papers  (2020-02-25T21:00:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.