Related papers: Towards trustworthy agentic AI: a comprehensive survey of safety, robustness, privacy, and system security

Towards trustworthy agentic AI: a comprehensive survey of safety, robustness, privacy, and system security

URL: http://arxiv.org/abs/2605.23989v1
Date: Sun, 17 May 2026 10:26:37 GMT
Title: Towards trustworthy agentic AI: a comprehensive survey of safety, robustness, privacy, and system security
Abstract summary: Agentic AI systems execute complex tasks autonomously, but their multi-step trajectories introduce new failure modes that challenge trustworthiness.<n>This survey provides a focused examination of trustworthy agentic AI through two core dimensions that are critical for high-risk deployments.<n>For each dimension, we clarify key concepts, identify where risks emerge along the agent workflow, and summarize stage-targeted mitigation strategies.
Score: 57.35851886874902
License: http://creativecommons.org/licenses/by/4.0/
Abstract: Agentic AI systems -- Large Language Models (LLMs) augmented with planning, tool use, memory, and long-horizon interactions -- can execute complex tasks autonomously, but their multi-step trajectories introduce new failure modes that challenge trustworthiness. This survey provides a focused examination of trustworthy agentic AI through two core dimensions that are critical for high-risk deployments: Safety and Robustness, and Privacy and System Security. For each dimension, we clarify key concepts, identify where risks emerge along the agent workflow, and summarize stage-targeted mitigation strategies. Other trustworthiness aspects (value alignment, transparency, fairness, and accountability) are discussed as relevant context rather than parallel chapters. To support consistent comparison and deployment decisions, we consolidate evaluation into a unified metrics-and-benchmarks hub, emphasizing both outcome and process signals (e.g., constraint violations, trace completeness, and adversarial success rates) and offering scenario-to-metric guidance for release gating. We conclude by outlining open challenges such as self-evolving agents, runtime monitoring and verification, privacy-preserving personalization, and the trust-utility trade-off, and present a case study of real-world security failures in open-source agentic systems. Our goal is to serve as a practical reference for researchers and practitioners building trustworthy agentic systems in high-stakes environments.

Related papers

Towards Trustworthy Embodied Intelligence: A Systems Framework and Graded Trustworthiness Levels [53.63220028820581]
Embodied intelligence integrates learned perception and decision making with real-time computation, control, and physical interaction.<n>We define trustworthy embodied intelligence as the sustained capacity to execute specified tasks reliably.
arXiv Detail & Related papers (2026-07-28T17:50:44Z)
Engineering Trustworthy Agentic AI for Critical Systems [1.8015174793525714]
The study adopts a trustworthiness model organized around five cross-cutting dimensions: safety and constraint satisfaction; robustness and reliability; transparency and interpretability; accountability and auditability; and privacy and security.<n>Building on this foundation, agentic systems architectures, threats, concrete trust mechanisms, and quantitative metrics are surveyed for direct application in agentic systems development and evaluation.<n>These principles are then examined across four constraint-bound engineering domains: power systems, autonomous vehicles/robotics/UAVs, high-performance computing, and communication networks.
arXiv Detail & Related papers (2026-07-20T22:28:16Z)
AgentCanary: A Security Evaluation Framework for Autonomous AI Agents in Real Executable Environments [32.982317538632806]
We propose AgentCanary, a comprehensive security evaluation framework for autonomous AI agents.<n>AgentCanary provides a systematic solution along three contributions.<n>We evaluate a broad set of frontier models on AgentCanary against multiple established adversarial attack methods.
arXiv Detail & Related papers (2026-06-09T06:55:37Z)
Safety in Embodied AI: A Survey of Risks, Attacks, and Defenses [168.50301366360344]
Embodied Artificial Intelligence (Embodied AI) integrates perception, cognition, planning, and interaction into agents that operate in safety-critical environments.<n>Unlike digital AI systems, embodied agents must act under uncertain sensing, incomplete knowledge, and dynamic human-robot interactions.<n>This survey provides a comprehensive review of safety research in embodied AI, examining attacks and defenses across the full embodied pipeline.
arXiv Detail & Related papers (2026-03-28T13:21:44Z)
Security Considerations for Artificial Intelligence Agents [7.055090485438426]
This article details Perplexity's observations and recommendations concerning the security of frontier AI agents.<n>We map attack surfaces across tools, connectors, hosting boundaries, and multi-agent coordination.<n>We identify standards and research gaps, guidance for secure multi-agent system design aligned with NIST risk management principles.
arXiv Detail & Related papers (2026-03-12T17:49:39Z)
Simulating and Understanding Deceptive Behaviors in Long-Horizon Interactions [18.182800471968132]
We introduce the first simulation framework for probing and evaluating deception in large language models.<n>We conduct experiments across 11 frontier models, spanning both closed and open-source systems.<n>We find that deception is model-dependent, increases with event pressure, and consistently erodes supervisor trust.
arXiv Detail & Related papers (2025-10-05T02:18:23Z)
TRiSM for Agentic AI: A Review of Trust, Risk, and Security Management in LLM-based Agentic Multi-Agent Systems [8.683314804719506]
This review presents a structured analysis of Trust, Risk, and Security Management (TRiSM) in the context of Agentic Multi-Agent Systems (AMAS)<n>We begin by examining the conceptual foundations of Agentic AI and highlight its architectural distinctions from traditional AI agents.<n>We then adapt and extend the AI TRiSM framework for Agentic AI, structured around key pillars: textit Explainability, ModelOps, Security, Privacy and textittheir lifecycle governance<n>A risk taxonomy is proposed to capture the unique threats and vulnerabilities of Agentic AI, ranging from coordination failures to
arXiv Detail & Related papers (2025-06-04T16:26:11Z)
LLM Agents Should Employ Security Principles [60.03651084139836]
This paper argues that the well-established design principles in information security should be employed when deploying Large Language Model (LLM) agents at scale.<n>We introduce AgentSandbox, a conceptual framework embedding these security principles to provide safeguards throughout an agent's life-cycle.
arXiv Detail & Related papers (2025-05-29T21:39:08Z)
Towards Trustworthy GUI Agents: A Survey [64.6445117343499]
This survey examines the trustworthiness of GUI agents in five critical dimensions.<n>We identify major challenges such as vulnerability to adversarial attacks, cascading failure modes in sequential decision-making.<n>As GUI agents become more widespread, establishing robust safety standards and responsible development practices is essential.
arXiv Detail & Related papers (2025-03-30T13:26:00Z)
AILuminate: Introducing v1.0 of the AI Risk and Reliability Benchmark from MLCommons [62.374792825813394]
This paper introduces AILuminate v1.0, the first comprehensive industry-standard benchmark for assessing AI-product risk and reliability.<n>The benchmark evaluates an AI system's resistance to prompts designed to elicit dangerous, illegal, or undesirable behavior in 12 hazard categories.
arXiv Detail & Related papers (2025-02-19T05:58:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.