Agent-SafetyBench: Evaluating the Safety of LLM Agents

• URL: http://arxiv.org/abs/2412.14470v1
• Date: Thu, 19 Dec 2024 02:35:15 GMT
• Title: Agent-SafetyBench: Evaluating the Safety of LLM Agents
• Authors: Zhexin Zhang, Shiyao Cui, Yida Lu, Jingzhuo Zhou, Junxiao Yang, Hongning Wang, Minlie Huang,
• Abstract summary: We introduce Agent-SafetyBench, a comprehensive benchmark to evaluate the safety of large language models (LLMs) Agent-SafetyBench encompasses 349 interaction environments and 2,000 test cases, evaluating 8 categories of safety risks and covering 10 common failure modes frequently encountered in unsafe interactions. Our evaluation of 16 popular LLM agents reveals a concerning result: none of the agents achieves a safety score above 60%.
• Score: 72.92604341646691
• License:
• Abstract: As large language models (LLMs) are increasingly deployed as agents, their integration into interactive environments and tool use introduce new safety challenges beyond those associated with the models themselves. However, the absence of comprehensive benchmarks for evaluating agent safety presents a significant barrier to effective assessment and further improvement. In this paper, we introduce Agent-SafetyBench, a comprehensive benchmark designed to evaluate the safety of LLM agents. Agent-SafetyBench encompasses 349 interaction environments and 2,000 test cases, evaluating 8 categories of safety risks and covering 10 common failure modes frequently encountered in unsafe interactions. Our evaluation of 16 popular LLM agents reveals a concerning result: none of the agents achieves a safety score above 60%. This highlights significant safety challenges in LLM agents and underscores the considerable need for improvement. Through quantitative analysis, we identify critical failure modes and summarize two fundamental safety detects in current LLM agents: lack of robustness and lack of risk awareness. Furthermore, our findings suggest that reliance on defense prompts alone is insufficient to address these safety issues, emphasizing the need for more advanced and robust strategies. We release Agent-SafetyBench at \url{https://github.com/thu-coai/Agent-SafetyBench} to facilitate further research and innovation in agent safety evaluation and improvement.

Related papers

• AGrail: A Lifelong Agent Guardrail with Effective and Adaptive Safety Detection [47.83354878065321]
  We propose AGrail, a lifelong guardrail to enhance agent safety. AGrail features adaptive safety check generation, effective safety check optimization, and tool compatibility and flexibility.
  arXiv  Detail & Related papers  (2025-02-17T05:12:33Z)
• SafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM Agents [42.69984822098671]
  We present SafeAgentBench -- a new benchmark for safety-aware task planning of embodied LLM agents. Best-performing baseline gets 69% success rate for safe tasks, but only 5% rejection rate for hazardous tasks.
  arXiv  Detail & Related papers  (2024-12-17T18:55:58Z)
• LabSafety Bench: Benchmarking LLMs on Safety Issues in Scientific Labs [80.45174785447136]
  Laboratory accidents pose significant risks to human life and property. Despite advancements in safety training, laboratory personnel may still unknowingly engage in unsafe practices. There is a growing concern about large language models (LLMs) for guidance in various fields.
  arXiv  Detail & Related papers  (2024-10-18T05:21:05Z)
• Multimodal Situational Safety [73.63981779844916]
  We present the first evaluation and analysis of a novel safety challenge termed Multimodal Situational Safety. For an MLLM to respond safely, whether through language or action, it often needs to assess the safety implications of a language query within its corresponding visual context. We develop the Multimodal Situational Safety benchmark (MSSBench) to assess the situational safety performance of current MLLMs.
  arXiv  Detail & Related papers  (2024-10-08T16:16:07Z)
• Athena: Safe Autonomous Agents with Verbal Contrastive Learning [3.102303947219617]
  Large language models (LLMs) have been utilized as language-based agents to perform a variety of tasks. In this study, we introduce the Athena framework which leverages the concept of verbal contrastive learning. The framework also incorporates a critiquing mechanism to guide the agent to prevent risky actions at every step.
  arXiv  Detail & Related papers  (2024-08-20T17:21:10Z)
• The Art of Defending: A Systematic Evaluation and Analysis of LLM Defense Strategies on Safety and Over-Defensiveness [56.174255970895466]
  Large Language Models (LLMs) play an increasingly pivotal role in natural language processing applications. This paper presents Safety and Over-Defensiveness Evaluation (SODE) benchmark.
  arXiv  Detail & Related papers  (2023-12-30T17:37:06Z)
• Safety Assessment of Chinese Large Language Models [51.83369778259149]
  Large language models (LLMs) may generate insulting and discriminatory content, reflect incorrect social values, and may be used for malicious purposes. To promote the deployment of safe, responsible, and ethical AI, we release SafetyPrompts including 100k augmented prompts and responses by LLMs.
  arXiv  Detail & Related papers  (2023-04-20T16:27:35Z)
• Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvements [76.80453043969209]
  This survey presents a framework for safety research pertaining to large models. We begin by introducing safety issues of wide concern, then delve into safety evaluation methods for large models. We explore the strategies for enhancing large model safety from training to deployment.
  arXiv  Detail & Related papers  (2023-02-18T09:32:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.