GuardAgent: Safeguard LLM Agents by a Guard Agent via Knowledge-Enabled Reasoning

• URL: http://arxiv.org/abs/2406.09187v1
• Date: Thu, 13 Jun 2024 14:49:26 GMT
• Title: GuardAgent: Safeguard LLM Agents by a Guard Agent via Knowledge-Enabled Reasoning
• Authors: Zhen Xiang, Linzhi Zheng, Yanjie Li, Junyuan Hong, Qinbin Li, Han Xie, Jiawei Zhang, Zidi Xiong, Chulin Xie, Carl Yang, Dawn Song, Bo Li,
• Abstract summary: Existing methods for enhancing the safety of large language models (LLMs) are not directly transferable to LLM-powered agents. We propose GuardAgent, the first LLM agent as a guardrail to other LLM agents. GuardAgent comprises two steps: 1) creating a task plan by analyzing the provided guard requests, and 2) generating guardrail code based on the task plan and executing the code by calling APIs or using external engines.
• Score: 79.07152553060601
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The rapid advancement of large language models (LLMs) has catalyzed the deployment of LLM-powered agents across numerous applications, raising new concerns regarding their safety and trustworthiness. Existing methods for enhancing the safety of LLMs are not directly transferable to LLM-powered agents due to their diverse objectives and output modalities. In this paper, we propose GuardAgent, the first LLM agent as a guardrail to other LLM agents. Specifically, GuardAgent oversees a target LLM agent by checking whether its inputs/outputs satisfy a set of given guard requests defined by the users. GuardAgent comprises two steps: 1) creating a task plan by analyzing the provided guard requests, and 2) generating guardrail code based on the task plan and executing the code by calling APIs or using external engines. In both steps, an LLM is utilized as the core reasoning component, supplemented by in-context demonstrations retrieved from a memory module. Such knowledge-enabled reasoning allows GuardAgent to understand various textual guard requests and accurately "translate" them into executable code that provides reliable guardrails. Furthermore, GuardAgent is equipped with an extendable toolbox containing functions and APIs and requires no additional LLM training, which underscores its generalization capabilities and low operational overhead. Additionally, we propose two novel benchmarks: an EICU-AC benchmark for assessing privacy-related access control for healthcare agents and a Mind2Web-SC benchmark for safety evaluation for web agents. We show the effectiveness of GuardAgent on these two benchmarks with 98.7% and 90.0% accuracy in moderating invalid inputs and outputs for the two types of agents, respectively. We also show that GuardAgent is able to define novel functions in adaption to emergent LLM agents and guard requests, which underscores its strong generalization capabilities.

Related papers

• AgentOccam: A Simple Yet Strong Baseline for LLM-Based Web Agents [52.13695464678006]
  This study enhances an LLM-based web agent by simply refining its observation and action space. AgentOccam surpasses the previous state-of-the-art and concurrent work by 9.8 (+29.4%) and 5.9 (+15.8%) absolute points respectively.
  arXiv  Detail & Related papers  (2024-10-17T17:50:38Z)
• AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents [84.96249955105777]
  LLM agents may pose a greater risk if misused, but their robustness remains underexplored. We propose a new benchmark called AgentHarm to facilitate research on LLM agent misuse. We find leading LLMs are surprisingly compliant with malicious agent requests without jailbreaking.
  arXiv  Detail & Related papers  (2024-10-11T17:39:22Z)
• AgentMonitor: A Plug-and-Play Framework for Predictive and Secure Multi-Agent Systems [43.333567687032904]
  AgentMonitor is a framework that integrates at the agent level to capture inputs and outputs, transforming them into statistics for training a regression model to predict task performance. It can further apply real-time corrections to address security risks posed by malicious agents, mitigating negative impacts and enhancing MAS security.
  arXiv  Detail & Related papers  (2024-08-27T11:24:38Z)
• BadAgent: Inserting and Activating Backdoor Attacks in LLM Agents [26.057916556444333]
  We show that such methods are vulnerable to our proposed backdoor attacks named BadAgent. Our proposed attack methods are extremely robust even after fine-tuning on trustworthy data.
  arXiv  Detail & Related papers  (2024-06-05T07:14:28Z)
• AgentLite: A Lightweight Library for Building and Advancing Task-Oriented LLM Agent System [91.41155892086252]
  We open-source a new AI agent library, AgentLite, which simplifies research investigation into LLM agents. AgentLite is a task-oriented framework designed to enhance the ability of agents to break down tasks. We introduce multiple practical applications developed with AgentLite to demonstrate its convenience and flexibility.
  arXiv  Detail & Related papers  (2024-02-23T06:25:20Z)
• Watch Out for Your Agents! Investigating Backdoor Threats to LLM-Based Agents [47.219047422240145]
  We take the first step to investigate one of the typical safety threats, backdoor attack, to LLM-based agents. Specifically, compared with traditional backdoor attacks on LLMs that are only able to manipulate the user inputs and model outputs, agent backdoor attacks exhibit more diverse and covert forms.
  arXiv  Detail & Related papers  (2024-02-17T06:48:45Z)
• TrustAgent: Towards Safe and Trustworthy LLM-based Agents [50.33549510615024]
  This paper presents an Agent-Constitution-based agent framework, TrustAgent, with a focus on improving the LLM-based agent safety. The proposed framework ensures strict adherence to the Agent Constitution through three strategic components: pre-planning strategy which injects safety knowledge to the model before plan generation, in-planning strategy which enhances safety during plan generation, and post-planning strategy which ensures safety by post-planning inspection.
  arXiv  Detail & Related papers  (2024-02-02T17:26:23Z)
• AgentTuning: Enabling Generalized Agent Abilities for LLMs [35.74502545364593]
  We present AgentTuning, a simple and general method to enhance the agent abilities of open large language models. We employ a hybrid instruction-tuning strategy by combining AgentInstruct with open-source instructions from general domains. Our evaluations show that AgentTuning enables LLMs' agent capabilities without compromising general abilities.
  arXiv  Detail & Related papers  (2023-10-19T15:19:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.