Imprompter: Tricking LLM Agents into Improper Tool Use

• URL: http://arxiv.org/abs/2410.14923v2
• Date: Tue, 22 Oct 2024 00:53:48 GMT
• Title: Imprompter: Tricking LLM Agents into Improper Tool Use
• Authors: Xiaohan Fu, Shuheng Li, Zihan Wang, Yihao Liu, Rajesh K. Gupta, Taylor Berg-Kirkpatrick, Earlence Fernandes,
• Abstract summary: Large Language Model (LLM) Agents are an emerging computing paradigm that blends generative machine learning with tools such as code interpreters, web browsing, email, and more generally, external resources. We contribute to the security foundations of agent-based systems and surface a new class of automatically computed obfuscated adversarial prompt attacks.
• Score: 35.255462653237885
• License:
• Abstract: Large Language Model (LLM) Agents are an emerging computing paradigm that blends generative machine learning with tools such as code interpreters, web browsing, email, and more generally, external resources. These agent-based systems represent an emerging shift in personal computing. We contribute to the security foundations of agent-based systems and surface a new class of automatically computed obfuscated adversarial prompt attacks that violate the confidentiality and integrity of user resources connected to an LLM agent. We show how prompt optimization techniques can find such prompts automatically given the weights of a model. We demonstrate that such attacks transfer to production-level agents. For example, we show an information exfiltration attack on Mistral's LeChat agent that analyzes a user's conversation, picks out personally identifiable information, and formats it into a valid markdown command that results in leaking that data to the attacker's server. This attack shows a nearly 80% success rate in an end-to-end evaluation. We conduct a range of experiments to characterize the efficacy of these attacks and find that they reliably work on emerging agent-based systems like Mistral's LeChat, ChatGLM, and Meta's Llama. These attacks are multimodal, and we show variants in the text-only and image domains.

Related papers

• Aligning LLMs to Be Robust Against Prompt Injection [55.07562650579068]
  We show that alignment can be a powerful tool to make LLMs more robust against prompt injection attacks. Our method -- SecAlign -- first builds an alignment dataset by simulating prompt injection attacks. Our experiments show that SecAlign robustifies the LLM substantially with a negligible hurt on model utility.
  arXiv  Detail & Related papers  (2024-10-07T19:34:35Z)
• On the Resilience of Multi-Agent Systems with Malicious Agents [58.79302663733702]
  This paper investigates what is the resilience of multi-agent system structures under malicious agents. We devise two methods, AutoTransform and AutoInject, to transform any agent into a malicious one. We show that two defense methods, introducing a mechanism for each agent to challenge others' outputs, or an additional agent to review and correct messages, can enhance system resilience.
  arXiv  Detail & Related papers  (2024-08-02T03:25:20Z)
• BadAgent: Inserting and Activating Backdoor Attacks in LLM Agents [26.057916556444333]
  We show that such methods are vulnerable to our proposed backdoor attacks named BadAgent. Our proposed attack methods are extremely robust even after fine-tuning on trustworthy data.
  arXiv  Detail & Related papers  (2024-06-05T07:14:28Z)
• Are you still on track!? Catching LLM Task Drift with Activations [55.75645403965326]
  Task drift allows attackers to exfiltrate data or influence the LLM's output for other users. We show that a simple linear classifier can detect drift with near-perfect ROC AUC on an out-of-distribution test set. We observe that this approach generalizes surprisingly well to unseen task domains, such as prompt injections, jailbreaks, and malicious instructions.
  arXiv  Detail & Related papers  (2024-06-02T16:53:21Z)
• InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents [3.5248694676821484]
  We introduce InjecAgent, a benchmark designed to assess the vulnerability of tool-integrated LLM agents to IPI attacks. InjecAgent comprises 1,054 test cases covering 17 different user tools and 62 attacker tools. We show that agents are vulnerable to IPI attacks, with ReAct-prompted GPT-4 vulnerable to attacks 24% of the time.
  arXiv  Detail & Related papers  (2024-03-05T06:21:45Z)
• KwaiAgents: Generalized Information-seeking Agent System with Large Language Models [33.59597020276034]
  Humans excel in critical thinking, planning, reflection, and harnessing available tools to interact with and interpret the world. Recent advancements in large language models (LLMs) suggest that machines might also possess the aforementioned human-like capabilities. We introduce KwaiAgents, a generalized information-seeking agent system based on LLMs.
  arXiv  Detail & Related papers  (2023-12-08T08:11:11Z)
• On Generative Agents in Recommendation [58.42840923200071]
  Agent4Rec is a user simulator in recommendation based on Large Language Models. Each agent interacts with personalized recommender models in a page-by-page manner.
  arXiv  Detail & Related papers  (2023-10-16T06:41:16Z)
• Misusing Tools in Large Language Models With Visual Adversarial Examples [34.82432122637917]
  We show that an attacker can use visual adversarial examples to cause attacker-desired tool usage. For example, the attacker could cause a victim LLM to delete calendar events, leak private conversations and book hotels. We construct these attacks using gradient-based adversarial training and characterize performance along multiple dimensions.
  arXiv  Detail & Related papers  (2023-10-04T22:10:01Z)
• Realistic simulation of users for IT systems in cyber ranges [63.20765930558542]
  We instrument each machine by means of an external agent to generate user activity. This agent combines both deterministic and deep learning based methods to adapt to different environment. We also propose conditional text generation models to facilitate the creation of conversations and documents.
  arXiv  Detail & Related papers  (2021-11-23T10:53:29Z)
• Attribution of Gradient Based Adversarial Attacks for Reverse Engineering of Deceptions [16.23543028393521]
  We present two techniques that support automated identification and attribution of adversarial ML attack toolchains. To the best of our knowledge, this is the first approach to attribute gradient based adversarial attacks and estimate their parameters.
  arXiv  Detail & Related papers  (2021-03-19T19:55:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.