Adversarial Attacks on Multimodal Agents

• URL: http://arxiv.org/abs/2406.12814v1
• Date: Tue, 18 Jun 2024 17:32:48 GMT
• Title: Adversarial Attacks on Multimodal Agents
• Authors: Chen Henry Wu, Jing Yu Koh, Ruslan Salakhutdinov, Daniel Fried, Aditi Raghunathan,
• Abstract summary: Vision-enabled language models (VLMs) are now used to build autonomous multimodal agents capable of taking actions in real environments. We show that multimodal agents raise new safety risks, even though attacking agents is more challenging than prior attacks due to limited access to and knowledge about the environment.
• Score: 73.97379283655127
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-enabled language models (VLMs) are now used to build autonomous multimodal agents capable of taking actions in real environments. In this paper, we show that multimodal agents raise new safety risks, even though attacking agents is more challenging than prior attacks due to limited access to and knowledge about the environment. Our attacks use adversarial text strings to guide gradient-based perturbation over one trigger image in the environment: (1) our captioner attack attacks white-box captioners if they are used to process images into captions as additional inputs to the VLM; (2) our CLIP attack attacks a set of CLIP models jointly, which can transfer to proprietary VLMs. To evaluate the attacks, we curated VisualWebArena-Adv, a set of adversarial tasks based on VisualWebArena, an environment for web-based multimodal agent tasks. Within an L-infinity norm of $16/256$ on a single image, the captioner attack can make a captioner-augmented GPT-4V agent execute the adversarial goals with a 75% success rate. When we remove the captioner or use GPT-4V to generate its own captions, the CLIP attack can achieve success rates of 21% and 43%, respectively. Experiments on agents based on other VLMs, such as Gemini-1.5, Claude-3, and GPT-4o, show interesting differences in their robustness. Further analysis reveals several key factors contributing to the attack's success, and we also discuss the implications for defenses as well. Project page: https://chenwu.io/attack-agent Code and data: https://github.com/ChenWu98/agent-attack

Related papers

• AegisLLM: Scaling Agentic Systems for Self-Reflective Defense in LLM Security [74.22452069013289]
  AegisLLM is a cooperative multi-agent defense against adversarial attacks and information leakage. We show that scaling agentic reasoning system at test-time substantially enhances robustness without compromising model utility. Comprehensive evaluations across key threat scenarios, including unlearning and jailbreaking, demonstrate the effectiveness of AegisLLM.
  arXiv  Detail & Related papers  (2025-04-29T17:36:05Z)
• WASP: Benchmarking Web Agent Security Against Prompt Injection Attacks [36.97842000562324]
  A benchmark called WASP introduces realistic web agent hijacking objectives and an isolated environment to test them. Our evaluation shows that even AI agents backed by models with advanced reasoning capabilities are susceptible to low-effort human-written prompt injections. Agents begin executing the adversarial instruction between 16 and 86% of the time but only achieve the goal between 0 and 17% of the time.
  arXiv  Detail & Related papers  (2025-04-22T17:51:03Z)
• Manipulating Multimodal Agents via Cross-Modal Prompt Injection [34.35145839873915]
  We identify a critical yet previously overlooked security vulnerability in multimodal agents. We propose CrossInject, a novel attack framework in which attackers embed adversarial perturbations across multiple modalities. Our method outperforms existing injection attacks, achieving at least a +26.4% increase in attack success rates.
  arXiv  Detail & Related papers  (2025-04-19T16:28:03Z)
• AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories [59.214178488091584]
  We propose AgentRewardBench, the first benchmark to assess the effectiveness of LLM judges for evaluating web agents. Using our benchmark, we evaluate 12 LLM judges and find that no single LLM excels across all benchmarks. We also find that the rule-based evaluation used by common benchmarks tends to underreport the success rate of web agents.
  arXiv  Detail & Related papers  (2025-04-11T19:49:22Z)
• Get the Agents Drunk: Memory Perturbations in Autonomous Agent-based Recommender Systems [29.35591074298123]
  Large language model-based agents are increasingly used in recommender systems (Agent4RSs) to achieve personalized behavior modeling. To the best of our knowledge, how robust Agent4RSs are remains unexplored. We propose the first work to attack Agent4RSs by perturbing agents' memories, not only to uncover their limitations but also to enhance their security and robustness.
  arXiv  Detail & Related papers  (2025-03-31T07:35:40Z)
• Imprompter: Tricking LLM Agents into Improper Tool Use [35.255462653237885]
  Large Language Model (LLM) Agents are an emerging computing paradigm that blends generative machine learning with tools such as code interpreters, web browsing, email, and more generally, external resources. We contribute to the security foundations of agent-based systems and surface a new class of automatically computed obfuscated adversarial prompt attacks.
  arXiv  Detail & Related papers  (2024-10-19T01:00:57Z)
• Agent-as-a-Judge: Evaluate Agents with Agents [61.33974108405561]
  We introduce the Agent-as-a-Judge framework, wherein agentic systems are used to evaluate agentic systems. This is an organic extension of the LLM-as-a-Judge framework, incorporating agentic features that enable intermediate feedback for the entire task-solving process. We present DevAI, a new benchmark of 55 realistic automated AI development tasks.
  arXiv  Detail & Related papers  (2024-10-14T17:57:02Z)
• Derail Yourself: Multi-turn LLM Jailbreak Attack through Self-discovered Clues [88.96201324719205]
  This study exposes the safety vulnerabilities of Large Language Models (LLMs) in multi-turn interactions. We introduce ActorAttack, a novel multi-turn attack method inspired by actor-network theory.
  arXiv  Detail & Related papers  (2024-10-14T16:41:49Z)
• AnyAttack: Towards Large-scale Self-supervised Generation of Targeted Adversarial Examples for Vision-Language Models [41.044385916368455]
  Vision-Language Models (VLMs) are vulnerable to image-based adversarial attacks. We propose AnyAttack, a self-supervised framework that generates targeted adversarial images for VLMs without label supervision.
  arXiv  Detail & Related papers  (2024-10-07T09:45:18Z)
• On the Resilience of LLM-Based Multi-Agent Collaboration with Faulty Agents [58.79302663733703]
  Large language model-based multi-agent systems have shown great abilities across various tasks due to the collaboration of expert agents. However, the impact of clumsy or even malicious agents, on the overall performance of the system remains underexplored. This paper investigates what is the resilience of various system structures under faulty agents.
  arXiv  Detail & Related papers  (2024-08-02T03:25:20Z)
• AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents [27.701301913159067]
  We introduce AgentDojo, an evaluation framework for agents that execute tools over untrusted data. AgentDojo is not a static test suite, but rather an environment for designing and evaluating new agent tasks, defenses, and adaptive attacks. We populate AgentDojo with 97 realistic tasks, 629 security test cases, and various attack and defense paradigms from the literature.
  arXiv  Detail & Related papers  (2024-06-19T08:55:56Z)
• White-box Multimodal Jailbreaks Against Large Vision-Language Models [61.97578116584653]
  We propose a more comprehensive strategy that jointly attacks both text and image modalities to exploit a broader spectrum of vulnerability within Large Vision-Language Models. Our attack method begins by optimizing an adversarial image prefix from random noise to generate diverse harmful responses in the absence of text input. An adversarial text suffix is integrated and co-optimized with the adversarial image prefix to maximize the probability of eliciting affirmative responses to various harmful instructions.
  arXiv  Detail & Related papers  (2024-05-28T07:13:30Z)
• VL-Trojan: Multimodal Instruction Backdoor Attacks against Autoregressive Visual Language Models [65.23688155159398]
  Autoregressive Visual Language Models (VLMs) showcase impressive few-shot learning capabilities in a multimodal context. Recently, multimodal instruction tuning has been proposed to further enhance instruction-following abilities. Adversaries can implant a backdoor by injecting poisoned samples with triggers embedded in instructions or images. We propose a multimodal instruction backdoor attack, namely VL-Trojan.
  arXiv  Detail & Related papers  (2024-02-21T14:54:30Z)
• Shadowcast: Stealthy Data Poisoning Attacks Against Vision-Language Models [73.37389786808174]
  This study takes the first step in exposing Vision-Language Models' susceptibility to data poisoning attacks. We introduce Shadowcast, a stealthy data poisoning attack where poison samples are visually indistinguishable from benign images. We show that Shadowcast effectively achieves the attacker's intentions using as few as 50 poison samples.
  arXiv  Detail & Related papers  (2024-02-05T18:55:53Z)
• Vision-LLMs Can Fool Themselves with Self-Generated Typographic Attacks [62.34019142949628]
  Typographic Attacks, which involve pasting misleading text onto an image, were noted to harm the performance of Vision-Language Models like CLIP. We introduce two novel and more effective textitSelf-Generated attacks which prompt the LVLM to generate an attack against itself. Using our benchmark, we uncover that Self-Generated attacks pose a significant threat, reducing LVLM(s) classification performance by up to 33%.
  arXiv  Detail & Related papers  (2024-02-01T14:41:20Z)
• InstructTA: Instruction-Tuned Targeted Attack for Large Vision-Language Models [13.21813503235793]
  Large vision-language models (LVLMs) have demonstrated their incredible capability in image understanding and response generation. In this paper, we formulate a novel and practical targeted attack scenario that the adversary can only know the vision encoder of the victim LVLM. We propose an instruction-tuned targeted attack (dubbed textscInstructTA) to deliver the targeted adversarial attack on LVLMs with high transferability.
  arXiv  Detail & Related papers  (2023-12-04T13:40:05Z)
• Malicious Agent Detection for Robust Multi-Agent Collaborative Perception [52.261231738242266]
  Multi-agent collaborative (MAC) perception is more vulnerable to adversarial attacks than single-agent perception. We propose Malicious Agent Detection (MADE), a reactive defense specific to MAC perception. We conduct comprehensive evaluations on a benchmark 3D dataset V2X-sim and a real-road dataset DAIR-V2X.
  arXiv  Detail & Related papers  (2023-10-18T11:36:42Z)
• How Robust is Google's Bard to Adversarial Image Attacks? [45.92999116520135]
  Multimodal Large Language Models (MLLMs) that integrate text and other modalities (especially vision) have achieved unprecedented performance in various multimodal tasks. However, due to the unsolved adversarial robustness problem of vision models, MLLMs can have more severe safety and security risks. We study the adversarial robustness of Google's Bard to better understand the vulnerabilities of commercial MLLMs.
  arXiv  Detail & Related papers  (2023-09-21T03:24:30Z)
• Image Hijacks: Adversarial Images can Control Generative Models at Runtime [8.603201325413192]
  We discover image hijacks, adversarial images that control the behaviour of vision-language models at inference time. We derive the Prompt Matching method, allowing us to train hijacks matching the behaviour of an arbitrary user-defined text prompt. We use Behaviour Matching to craft hijacks for four types of attack, forcing VLMs to generate outputs of the adversary's choice, leak information from their context window, override their safety training, and believe false statements.
  arXiv  Detail & Related papers  (2023-09-01T03:53:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.