Towards Action Hijacking of Large Language Model-based Agent

• URL: http://arxiv.org/abs/2412.10807v1
• Date: Sat, 14 Dec 2024 12:11:26 GMT
• Title: Towards Action Hijacking of Large Language Model-based Agent
• Authors: Yuyang Zhang, Kangjie Chen, Xudong Jiang, Yuxiang Sun, Run Wang, Lina Wang,
• Abstract summary: We introduce Name, a novel hijacking attack to manipulate the action plans of black-box agent system. Our approach achieved an average bypass rate of 92.7% for safety filters.
• Score: 39.19067800226033
• License:
• Abstract: In the past few years, intelligent agents powered by large language models (LLMs) have achieved remarkable progress in performing complex tasks. These LLM-based agents receive queries as tasks and decompose them into various subtasks via the equipped LLMs to guide the action of external entities (\eg{}, tools, AI-agents) to answer the questions from users. Empowered by their exceptional capabilities of understanding and problem-solving, they are widely adopted in labor-intensive sectors including healthcare, finance, code completion, \etc{} At the same time, there are also concerns about the potential misuse of these agents, prompting the built-in safety guards from service providers. To circumvent the built-in guidelines, the prior studies proposed a multitude of attacks including memory poisoning, jailbreak, and prompt injection. These studies often fail to maintain effectiveness across safety filters employed by agents due to the restricted privileges and the harmful semantics in queries. In this paper, we introduce \Name, a novel hijacking attack to manipulate the action plans of black-box agent system. \Name first collects the action-aware memory through prompt theft from long-term memory. It then leverages the internal memory retrieval mechanism of the agent to provide an erroneous context. The huge gap between the latent spaces of the retriever and safety filters allows our method to bypass the detection easily. Extensive experimental results demonstrate the effectiveness of our apporach (\eg{}, 99.67\% ASR). Besides, our approach achieved an average bypass rate of 92.7\% for safety filters.

Related papers

• Unveiling Privacy Risks in LLM Agent Memory [40.26158509307175]
  Large Language Model (LLM) agents have become increasingly prevalent across various real-world applications. They enhance decision-making by storing private user-agent interactions in the memory module for demonstrations. We propose a Memory EXTRaction Attack (MEXTRA) to extract private information from memory.
  arXiv  Detail & Related papers  (2025-02-17T19:55:53Z)
• The Task Shield: Enforcing Task Alignment to Defend Against Indirect Prompt Injection in LLM Agents [6.829628038851487]
  Large Language Model (LLM) agents are increasingly being deployed as conversational assistants capable of performing complex real-world tasks through tool integration. In particular, indirect prompt injection attacks pose a critical threat, where malicious instructions embedded within external data sources can manipulate agents to deviate from user intentions. We propose a novel perspective that reframes agent security from preventing harmful actions to ensuring task alignment, requiring every agent action to serve user objectives.
  arXiv  Detail & Related papers  (2024-12-21T16:17:48Z)
• PrivAgent: Agentic-based Red-teaming for LLM Privacy Leakage [78.33839735526769]
  LLMs may be fooled into outputting private information under carefully crafted adversarial prompts. PrivAgent is a novel black-box red-teaming framework for privacy leakage.
  arXiv  Detail & Related papers  (2024-12-07T20:09:01Z)
• Targeting the Core: A Simple and Effective Method to Attack RAG-based Agents via Direct LLM Manipulation [4.241100280846233]
  AI agents, powered by large language models (LLMs), have transformed human-computer interactions by enabling seamless, natural, and context-aware communication. This paper investigates a critical vulnerability: adversarial attacks targeting the LLM core within AI agents.
  arXiv  Detail & Related papers  (2024-12-05T18:38:30Z)
• Compromising Embodied Agents with Contextual Backdoor Attacks [69.71630408822767]
  Large language models (LLMs) have transformed the development of embodied intelligence. This paper uncovers a significant backdoor security threat within this process. By poisoning just a few contextual demonstrations, attackers can covertly compromise the contextual environment of a black-box LLM.
  arXiv  Detail & Related papers  (2024-08-06T01:20:12Z)
• Breaking Agents: Compromising Autonomous LLM Agents Through Malfunction Amplification [35.16099878559559]
  Large language models (LLMs) have experienced significant development and are being deployed in real-world applications. We introduce a new type of attack that causes malfunctions by misleading the agent into executing repetitive or irrelevant actions. Our experiments reveal that these attacks can induce failure rates exceeding 80% in multiple scenarios.
  arXiv  Detail & Related papers  (2024-07-30T14:35:31Z)
• AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases [73.04652687616286]
  We propose AgentPoison, the first backdoor attack targeting generic and RAG-based LLM agents by poisoning their long-term memory or RAG knowledge base. Unlike conventional backdoor attacks, AgentPoison requires no additional model training or fine-tuning. On each agent, AgentPoison achieves an average attack success rate higher than 80% with minimal impact on benign performance.
  arXiv  Detail & Related papers  (2024-07-17T17:59:47Z)
• GuardAgent: Safeguard LLM Agents by a Guard Agent via Knowledge-Enabled Reasoning [79.07152553060601]
  Existing methods for enhancing the safety of large language models (LLMs) are not directly transferable to LLM-powered agents. We propose GuardAgent, the first LLM agent as a guardrail to other LLM agents. GuardAgent comprises two steps: 1) creating a task plan by analyzing the provided guard requests, and 2) generating guardrail code based on the task plan and executing the code by calling APIs or using external engines.
  arXiv  Detail & Related papers  (2024-06-13T14:49:26Z)
• On Prompt-Driven Safeguarding for Large Language Models [172.13943777203377]
  We find that in the representation space, the input queries are typically moved by safety prompts in a "higher-refusal" direction. Inspired by these findings, we propose a method for safety prompt optimization, namely DRO. Treating a safety prompt as continuous, trainable embeddings, DRO learns to move the queries' representations along or opposite the refusal direction, depending on their harmfulness.
  arXiv  Detail & Related papers  (2024-01-31T17:28:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.