MIRAGE: Context-Aware Prompt Injection against Mobile GUI Agents via User-Generated Content

• URL: http://arxiv.org/abs/2605.28116v1
• Date: Wed, 27 May 2026 08:09:53 GMT
• Title: MIRAGE: Context-Aware Prompt Injection against Mobile GUI Agents via User-Generated Content
• Abstract summary: We present MIRAGE, a pipeline that turns benign mobile screenshots into prompt-injection samples.<n>A key challenge is that an injected screenshot must stay visually indistinguishable from genuine user content while still diverting the agent.<n>On a 1,111-sample benchmark spanning ten applications and eleven attack intents, all five evaluated VLM agents are vulnerable.
• Score: 32.239107706539464
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Mobile graphical user interface (GUI) agents driven by vision-language models (VLMs) perceive the screen as rendered pixels and choose actions from what they see, so they cannot reliably separate trusted interface elements from user-generated content. We present MIRAGE (Mobile Injection of Realistic Adversarial GUI Examples), a pipeline that turns benign mobile screenshots into prompt-injection samples by placing attacker-controlled text into ordinary user-generated content regions, without modifying the agent, the application, or the operating system. MIRAGE operates in three stages: a Localizer identifies user-controllable regions on the screenshot, a Generator synthesises context-aware payloads and renders them in the application's native style, and a Curator moderates realism and balances the samples across applications, region types, and attack intents. A key challenge is that an injected screenshot must stay visually indistinguishable from genuine user content while still diverting the agent; we address this by separating the stages that control reach, realism, and distributional balance. On a 1,111-sample benchmark spanning ten applications and eleven attack intents, all five evaluated VLM agents are vulnerable, with attack success rates of 23%-30%, and MIRAGE scores higher on human realism ratings than the strongest prior attack (3.02 versus 2.52 out of 5). We further find that per-sample realism and attack success are uncorrelated, so visual-quality filtering alone cannot reliably defend against this threat.

Related papers

• SIR: Self-improving Red-teaming for Compute Use Agents [71.71987044117371]
  Computer use agents (CUAs) are vision-language models that perceive a screen and act on a real operating system through mouse, keyboard, and terminal.<n>Because they can be exposed to untrusted content while operating, they are vulnerable to indirect prompt injection (IPI)<n>We present SIR, a black box IPI attack that composes stealthy injections from a small library of reusable principles stated in plain language.
  arXiv  Detail & Related papers  (2026-08-31T03:39:43Z)
• ALLUDE: A Unified Evaluation System for Configurable Attacks in Differentiable Environments [57.11645440673911]
  Adrial attacks against vision models like object detectors are often evaluated under limited conditions, leaving their performance under-characterized.<n>We present ALLUDE, which offers first-of-its-kind evaluation capabilities across Linux and Windows.
  arXiv  Detail & Related papers  (2026-07-19T05:02:58Z)
• (A)I Sees What You Don't: Exploiting New Attack Surfaces in Third-Party Mobile Agents [6.848663048236236]
  Third-party mobile agents powered by Vision-Language Models (VLMs) have emerged as a promising paradigm for automating smartphone interactions.<n>In this paper, we summarize key differences between third-party mobile agent apps and general apps when interacting with the environment, analyze the security posture of agents, and identify two unique attack surfaces.<n>We design and implement seven concrete attacks, from subliminal text injection and invisible pixel zone exploitation to screenshot tampering and host PC command injection.
  arXiv  Detail & Related papers  (2026-07-01T02:17:19Z)
• Who Pays the Price? Stakeholder-Centric Prompt Injection Benchmarking for Real-world Web Agents [93.19140872946842]
  Web agents driven by large language models (LLMs) are increasingly deployed in real-world environments.<n>This makes them vulnerable to prompt-injection attacks, in which seemingly benign content embeds adversarial instructions that manipulate agent behaviour.<n>We introduce textbfsysname, a benchmark to systematically categorize and attribute harm in real-world web agent systems.
  arXiv  Detail & Related papers  (2026-06-11T14:12:43Z)
• AgentRAE: Remote Action Execution through Notification-based Visual Backdoors against Screenshots-based Mobile GUI Agents [18.82273534480229]
  Mobile graphical user interface (GUI) agents autonomously control applications and operating systems (OS)<n>We propose AgentRAE, a novel backdoor attack capable of inducing Remote Action Execution in mobile GUI agents using visually natural triggers.<n>Our evaluation reveals that the proposed backdoor preserves clean performance with an attack success rate of over 90% across ten mobile operations.
  arXiv  Detail & Related papers  (2026-03-24T09:51:43Z)
• GhostEI-Bench: Do Mobile Agents Resilience to Environmental Injection in Dynamic On-Device Environments? [30.170538068791263]
  Vision-Language Models (VLMs) are increasingly deployed as autonomous agents to navigate mobile graphical user interfaces (GUIs)<n>Environment injection corrupts an agent's visual perception by inserting adversarial UI elements directly into the GUI.<n>GhostEI-Bench is the first benchmark for assessing mobile agents under environmental injection attacks within dynamic, executable environments.
  arXiv  Detail & Related papers  (2025-10-23T08:33:24Z)
• Effective and Stealthy One-Shot Jailbreaks on Deployed Mobile Vision-Language Agents [29.62914440645731]
  We present a one-shot jailbreak attack that leverages in-app prompt injections.<n> malicious apps embed short prompts in UI text that remain inert during human interaction but are revealed when an agent drives the UI via ADB.<n>Our framework comprises three crucial components: (1) low-privilege perception-chain targeting, which injects payloads into malicious apps as the agent's visual inputs; (2) user-invisible activation, a touch-based trigger that discriminates agent from human touches using physical touch attributes and exposes the payload only during agent operation; and (3) one-shot prompt efficacy, a stealthy-guided, character-level
  arXiv  Detail & Related papers  (2025-10-09T05:34:57Z)
• VisualTrap: A Stealthy Backdoor Attack on GUI Agents via Visual Grounding Manipulation [73.92237451442752]
  This work reveals that the visual grounding of GUI agent-mapping textual plans to GUI elements can introduce vulnerabilities.<n>With backdoor attack targeting visual grounding, the agent's behavior can be compromised even when given correct task-solving plans.<n>We propose VisualTrap, a method that can hijack the grounding by misleading the agent to locate textual plans to trigger locations instead of the intended targets.
  arXiv  Detail & Related papers  (2025-07-09T14:36:00Z)
• Hijacking JARVIS: Benchmarking Mobile GUI Agents against Unprivileged Third Parties [19.430061128447022]
  We present the first systematic investigation into the vulnerabilities of mobile GUI agents.<n>We introduce a scalable attack simulation framework AgentHazard, which enables flexible and targeted modifications of screen content.<n>We develop a benchmark suite comprising both a dynamic task execution environment and a static dataset of attack scenarios.<n>Our findings reveal that all examined agents are significantly influenced by misleading third-party content.
  arXiv  Detail & Related papers  (2025-07-06T03:31:36Z)
• Poison Once, Control Anywhere: Clean-Text Visual Backdoors in VLM-based Mobile Agents [54.35629963816521]
  This work introduces VIBMA, the first clean-text backdoor attack targeting VLM-based mobile agents.<n>The attack injects malicious behaviors into the model by modifying only the visual input.<n>We show that our attack achieves high success rates while preserving clean-task behavior.
  arXiv  Detail & Related papers  (2025-06-16T08:09:32Z)
• EVA: Red-Teaming GUI Agents via Evolving Indirect Prompt Injection [14.83331240126743]
  multimodal agents are increasingly trained to operate graphical user interfaces (GUIs) to complete user tasks.<n>We propose EVA, a framework for indirect prompt injection, which transforms the attack into a closed loop optimization.<n>We evaluate EVA on six widely used generalist and specialist GUI agents in realistic settings such as popup manipulation, chat based phishing, payments, and email composition.
  arXiv  Detail & Related papers  (2025-05-20T12:41:05Z)
• Dissecting Adversarial Robustness of Multimodal LM Agents [70.2077308846307]
  We manually create 200 targeted adversarial tasks and evaluation scripts in a realistic threat model on top of VisualWebArena.<n>We find that we can successfully break latest agents that use black-box frontier LMs, including those that perform reflection and tree search.<n>We also use ARE to rigorously evaluate how the robustness changes as new components are added.
  arXiv  Detail & Related papers  (2024-06-18T17:32:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.