Agentic AI Security: Threats, Defenses, Evaluation, and Open Challenges

• URL: http://arxiv.org/abs/2510.23883v1
• Date: Mon, 27 Oct 2025 21:48:11 GMT
• Title: Agentic AI Security: Threats, Defenses, Evaluation, and Open Challenges
• Authors: Shrestha Datta, Shahriar Kabir Nahin, Anshuman Chhabra, Prasant Mohapatra,
• Abstract summary: Agentic AI systems powered by large language models (LLMs) are emerging as powerful, flexible platforms for automation.<n>Their ability to autonomously execute tasks across web, software, and physical environments creates new and amplified security risks.<n>This survey outlines a taxonomy of threats specific to agentic AI, reviews recent benchmarks and evaluation methodologies, and discusses defense strategies.
• Score: 14.546961299604554
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Agentic AI systems powered by large language models (LLMs) and endowed with planning, tool use, memory, and autonomy, are emerging as powerful, flexible platforms for automation. Their ability to autonomously execute tasks across web, software, and physical environments creates new and amplified security risks, distinct from both traditional AI safety and conventional software security. This survey outlines a taxonomy of threats specific to agentic AI, reviews recent benchmarks and evaluation methodologies, and discusses defense strategies from both technical and governance perspectives. We synthesize current research and highlight open challenges, aiming to support the development of secure-by-design agent systems.

Related papers

• AI-Driven Cybersecurity Threats: A Survey of Emerging Risks and Defensive Strategies [0.0]
  This paper aims to analyze emerging risks, attack mechanisms, and defense shortcomings related to AI in cybersecurity.<n>We introduce a comparative taxonomy connecting AI capabilities with threat modalities and defenses.<n>Our findings emphasize the urgency for explainable, interdisciplinary, and regulatory-compliant AI defense systems.
  arXiv  Detail & Related papers  (2026-01-06T05:09:40Z)
• Securing Agentic AI Systems -- A Multilayer Security Framework [0.0]
  Securing Agentic Artificial Intelligence (AI) systems requires addressing the complex cyber risks introduced by autonomous, decision-making, and adaptive behaviors.<n>Existing AI security frameworks do not adequately address these challenges or the unique nuances of agentic AI.<n>This research develops a lifecycle-aware security framework specifically designed for agentic AI systems.
  arXiv  Detail & Related papers  (2025-12-19T20:22:25Z)
• Cisco Integrated AI Security and Safety Framework Report [3.162988913169078]
  This paper presents Cisco's Integrated AI Security and Safety Framework ("AI Security Framework")<n>The framework can be used to classify, integrate, and operationalize the full range of AI risks.<n>It integrates AI security and AI safety across modalities, agents, pipelines, and the broader ecosystem.
  arXiv  Detail & Related papers  (2025-12-15T02:12:12Z)
• A Systematic Survey of Model Extraction Attacks and Defenses: State-of-the-Art and Perspectives [65.3369988566853]
  Recent studies have demonstrated that adversaries can replicate a target model's functionality.<n>Model Extraction Attacks pose threats to intellectual property, privacy, and system security.<n>We propose a novel taxonomy that classifies MEAs according to attack mechanisms, defense approaches, and computing environments.
  arXiv  Detail & Related papers  (2025-08-20T19:49:59Z)
• Report on NSF Workshop on Science of Safe AI [75.96202715567088]
  New advances in machine learning are leading to new opportunities to develop technology-based solutions to societal problems.<n>To fulfill the promise of AI, we must address how to develop AI-based systems that are accurate and performant but also safe and trustworthy.<n>This report is the result of the discussions in the working groups that addressed different aspects of safety at the workshop.
  arXiv  Detail & Related papers  (2025-06-24T18:55:29Z)
• Offensive Security for AI Systems: Concepts, Practices, and Applications [0.0]
  Traditional defensive measures often fall short against the unique and evolving threats facing AI-driven technologies.<n>This paper emphasizes proactive threat simulation and adversarial testing to uncover vulnerabilities throughout the AI lifecycle.
  arXiv  Detail & Related papers  (2025-05-09T18:58:56Z)
• Towards Trustworthy GUI Agents: A Survey [64.6445117343499]
  This survey examines the trustworthiness of GUI agents in five critical dimensions.<n>We identify major challenges such as vulnerability to adversarial attacks, cascading failure modes in sequential decision-making.<n>As GUI agents become more widespread, establishing robust safety standards and responsible development practices is essential.
  arXiv  Detail & Related papers  (2025-03-30T13:26:00Z)
• Attack Atlas: A Practitioner's Perspective on Challenges and Pitfalls in Red Teaming GenAI [52.138044013005]
  generative AI, particularly large language models (LLMs), become increasingly integrated into production applications. New attack surfaces and vulnerabilities emerge and put a focus on adversarial threats in natural language and multi-modal systems. Red-teaming has gained importance in proactively identifying weaknesses in these systems, while blue-teaming works to protect against such adversarial attacks. This work aims to bridge the gap between academic insights and practical security measures for the protection of generative AI systems.
  arXiv  Detail & Related papers  (2024-09-23T10:18:10Z)
• EARBench: Towards Evaluating Physical Risk Awareness for Task Planning of Foundation Model-based Embodied AI Agents [53.717918131568936]
  Embodied artificial intelligence (EAI) integrates advanced AI models into physical entities for real-world interaction.<n>Foundation models as the "brain" of EAI agents for high-level task planning have shown promising results.<n>However, the deployment of these agents in physical environments presents significant safety challenges.<n>This study introduces EARBench, a novel framework for automated physical risk assessment in EAI scenarios.
  arXiv  Detail & Related papers  (2024-08-08T13:19:37Z)
• AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathways [10.16690494897609]
  An Artificial Intelligence (AI) agent is a software entity that autonomously performs tasks or makes decisions based on pre-defined objectives and data inputs. This survey delves into the emerging security threats faced by AI agents, categorizing them into four critical knowledge gaps. By systematically reviewing these threats, this paper highlights both the progress made and the existing limitations in safeguarding AI agents.
  arXiv  Detail & Related papers  (2024-06-04T01:22:31Z)
• Asset-centric Threat Modeling for AI-based Systems [7.696807063718328]
  This paper presents ThreatFinderAI, an approach and tool to model AI-related assets, threats, countermeasures, and quantify residual risks. To evaluate the practicality of the approach, participants were tasked to recreate a threat model developed by cybersecurity experts of an AI-based healthcare platform. Overall, the solution's usability was well-perceived and effectively supports threat identification and risk discussion.
  arXiv  Detail & Related papers  (2024-03-11T08:40:01Z)
• Managing extreme AI risks amid rapid progress [171.05448842016125]
  We describe risks that include large-scale social harms, malicious uses, and irreversible loss of human control over autonomous AI systems. There is a lack of consensus about how exactly such risks arise, and how to manage them. Present governance initiatives lack the mechanisms and institutions to prevent misuse and recklessness, and barely address autonomous systems.
  arXiv  Detail & Related papers  (2023-10-26T17:59:06Z)
• Attacks, Defenses, And Tools: A Framework To Facilitate Robust AI/ML Systems [2.5137859989323528]
  Software systems are increasingly relying on Artificial Intelligence (AI) and Machine Learning (ML) components. This paper presents a framework to characterize attacks and weaknesses associated with AI-enabled systems.
  arXiv  Detail & Related papers  (2022-02-18T22:54:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.