Attacking Vision-Language Computer Agents via Pop-ups

• URL: http://arxiv.org/abs/2411.02391v1
• Date: Mon, 04 Nov 2024 18:56:42 GMT
• Title: Attacking Vision-Language Computer Agents via Pop-ups
• Authors: Yanzhe Zhang, Tao Yu, Diyi Yang,
• Abstract summary: We show that VLM agents can be easily attacked by a set of carefully designed adversarial pop-ups. This distraction leads agents to click these pop-ups instead of performing the tasks as usual.
• Score: 61.744008541021124
• License:
• Abstract: Autonomous agents powered by large vision and language models (VLM) have demonstrated significant potential in completing daily computer tasks, such as browsing the web to book travel and operating desktop software, which requires agents to understand these interfaces. Despite such visual inputs becoming more integrated into agentic applications, what types of risks and attacks exist around them still remain unclear. In this work, we demonstrate that VLM agents can be easily attacked by a set of carefully designed adversarial pop-ups, which human users would typically recognize and ignore. This distraction leads agents to click these pop-ups instead of performing the tasks as usual. Integrating these pop-ups into existing agent testing environments like OSWorld and VisualWebArena leads to an attack success rate (the frequency of the agent clicking the pop-ups) of 86% on average and decreases the task success rate by 47%. Basic defense techniques such as asking the agent to ignore pop-ups or including an advertisement notice, are ineffective against the attack.

Related papers

• AdvWeb: Controllable Black-box Attacks on VLM-powered Web Agents [22.682464365220916]
  AdvWeb is a novel black-box attack framework designed against web agents. We train and optimize the adversarial prompter model using DPO. Unlike prior approaches, our adversarial string injection maintains stealth and control.
  arXiv  Detail & Related papers  (2024-10-22T20:18:26Z)
• Imprompter: Tricking LLM Agents into Improper Tool Use [35.255462653237885]
  Large Language Model (LLM) Agents are an emerging computing paradigm that blends generative machine learning with tools such as code interpreters, web browsing, email, and more generally, external resources. We contribute to the security foundations of agent-based systems and surface a new class of automatically computed obfuscated adversarial prompt attacks.
  arXiv  Detail & Related papers  (2024-10-19T01:00:57Z)
• AgentDojo: A Dynamic Environment to Evaluate Attacks and Defenses for LLM Agents [27.701301913159067]
  We introduce AgentDojo, an evaluation framework for agents that execute tools over untrusted data. AgentDojo is not a static test suite, but rather an environment for designing and evaluating new agent tasks, defenses, and adaptive attacks. We populate AgentDojo with 97 realistic tasks, 629 security test cases, and various attack and defense paradigms from the literature.
  arXiv  Detail & Related papers  (2024-06-19T08:55:56Z)
• Adversarial Attacks on Multimodal Agents [73.97379283655127]
  Vision-enabled language models (VLMs) are now used to build autonomous multimodal agents capable of taking actions in real environments. We show that multimodal agents raise new safety risks, even though attacking agents is more challenging than prior attacks due to limited access to and knowledge about the environment.
  arXiv  Detail & Related papers  (2024-06-18T17:32:48Z)
• InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents [3.5248694676821484]
  We introduce InjecAgent, a benchmark designed to assess the vulnerability of tool-integrated LLM agents to IPI attacks. InjecAgent comprises 1,054 test cases covering 17 different user tools and 62 attacker tools. We show that agents are vulnerable to IPI attacks, with ReAct-prompted GPT-4 vulnerable to attacks 24% of the time.
  arXiv  Detail & Related papers  (2024-03-05T06:21:45Z)
• WIPI: A New Web Threat for LLM-Driven Web Agents [28.651763099760664]
  We introduce a novel threat, WIPI, that indirectly controls Web Agent to execute malicious instructions embedded in publicly accessible webpages. To launch a successful WIPI works in a black-box environment. Our methodology achieves an average attack success rate (ASR) exceeding 90% even in pure black-box scenarios.
  arXiv  Detail & Related papers  (2024-02-26T19:01:54Z)
• Watch Out for Your Agents! Investigating Backdoor Threats to LLM-Based Agents [47.219047422240145]
  We take the first step to investigate one of the typical safety threats, backdoor attack, to LLM-based agents. Specifically, compared with traditional backdoor attacks on LLMs that are only able to manipulate the user inputs and model outputs, agent backdoor attacks exhibit more diverse and covert forms.
  arXiv  Detail & Related papers  (2024-02-17T06:48:45Z)
• Pre-trained Trojan Attacks for Visual Recognition [106.13792185398863]
  Pre-trained vision models (PVMs) have become a dominant component due to their exceptional performance when fine-tuned for downstream tasks. We propose the Pre-trained Trojan attack, which embeds backdoors into a PVM, enabling attacks across various downstream vision tasks. We highlight the challenges posed by cross-task activation and shortcut connections in successful backdoor attacks.
  arXiv  Detail & Related papers  (2023-12-23T05:51:40Z)
• Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection [64.67495502772866]
  Large Language Models (LLMs) are increasingly being integrated into various applications. We show how attackers can override original instructions and employed controls using Prompt Injection attacks. We derive a comprehensive taxonomy from a computer security perspective to systematically investigate impacts and vulnerabilities.
  arXiv  Detail & Related papers  (2023-02-23T17:14:38Z)
• Adversarial Attacks On Multi-Agent Communication [80.4392160849506]
  Modern autonomous systems will soon be deployed at scale, opening up the possibility for cooperative multi-agent systems. Such advantages rely heavily on communication channels which have been shown to be vulnerable to security breaches. In this paper, we explore such adversarial attacks in a novel multi-agent setting where agents communicate by sharing learned intermediate representations.
  arXiv  Detail & Related papers  (2021-01-17T00:35:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.