PriMobiBench: Characterizing Visual Privacy Leakage in VLM-Driven Mobile GUI Agents

• URL: http://arxiv.org/abs/2609.13873v1
• Date: Sat, 12 Sep 2026 10:55:54 GMT
• Title: PriMobiBench: Characterizing Visual Privacy Leakage in VLM-Driven Mobile GUI Agents
• Abstract summary: We present PriMobiBench, the first benchmark for evaluating privacy leakage and visual profiling in screenshot-driven mobile agents.<n>We also introduce MobiLeak, a dataset of execution traces from 16 apps, covering 25 privacy attributes with 2,960 embedded privacy instances.
• Score: 18.88920135607137
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Mobile GUI agents increasingly rely on Vision-Language Models (VLMs) to automate smartphone tasks by interpreting screenshot streams. However, this design introduces serious and underexplored privacy risks, including direct leakage of sensitive on-screen information and unintended user profiling. The absence of standardized benchmarks makes it difficult to quantify these risks in realistic mobile agent workflows. To address this gap, we propose PriMobiBench, the first benchmark for systematically evaluating privacy leakage and visual profiling in screenshot-driven mobile agents. It provides a unified pipeline for data generation, agent trajectory construction, and multi-model evaluation. We also introduce MobiLeak, a dataset of execution traces from 16 apps, covering 25 privacy attributes with 2,960 embedded privacy instances. Our results reveal substantial risks: (1) VLMs can directly extract sensitive information with up to 82.5% success rate; (2) beyond explicit leakage, they can infer user profiles from aggregated visual evidence with approximately 70% success. We further propose a mitigation that masks privacy-sensitive but task-irrelevant UI elements before cloud processing, reducing profiling success by up to 58% with only approximately 8% performance loss. Overall, our work provides the first systematic benchmark for visual privacy risks in mobile GUI agents, demonstrates that both leakage and profiling are feasible at a highly concerning level, and offers a practical direction for mitigation.

Related papers

• CAPED: Context-Aware Privacy Exposure Defense for Mobile GUI Agents [6.4324284621588745]
  Screenshot-based mobile GUI agents can operate ordinary smartphone apps through the same visual interface as a human user.<n> screenshots may expose contacts, messages, photos, files, recommendations, health cues, and other sensitive context unrelated to the user's request.<n>This paper presents CAPED, a context-aware pre-upload exposure control layer for mobile GUI agents.
  arXiv  Detail & Related papers  (2026-06-10T20:48:39Z)
• Unveiling Privacy Risks in Multi-modal Large Language Models: Task-specific Vulnerabilities and Mitigation Challenges [55.425628316813174]
  Multi-modal Large Language Models (MLLMs) process both text and images.<n>Compared to text-only models, MLLMs can extract and expose sensitive information embedded in images.<n>Some MLLMs are susceptible to privacy breaches, leaking sensitive data embedded in images or stored in memory.
  arXiv  Detail & Related papers  (2026-06-08T07:19:42Z)
• Don't Trust Us: A privacy-by-design android malware detection pipeline [0.7009487789080343]
  Existing privacy-aware approaches typically enforce privacy after data collection.<n>We argue that this requirement should be removed rather than managed.<n>Android malware detection should be privacy-aware by design, so that effective analysis does not depend on sensitive data being accessed in the first place.
  arXiv  Detail & Related papers  (2026-06-02T14:36:31Z)
• PrivacyPeek: Auditing What LLM-Based Agents Acquire, Not Just What They Say [48.84133320567554]
  LLM-based agents are rapidly advancing, autonomously invoking external tools to complete tasks for users.<n> Existing privacy benchmarks audit what the agent's response or outgoing actions disclose, but overlook the acquisition stage where data first enters the agent's context.<n>We introduce emphPrivacyPeek, a benchmark for evaluating acquisition-stage privacy leakage of LLM-based agents.
  arXiv  Detail & Related papers  (2026-05-29T04:55:12Z)
• Do Phone-Use Agents Respect Your Privacy? [97.81424230136075]
  We introduce MyPhoneBench, a verifiable evaluation framework for privacy behavior in mobile agents.<n>We operationalize privacy-respecting phone use as permissioned access, minimal disclosure, and user-controlled memory through a minimal privacy contract.<n>Across five frontier models on 10 mobile apps and 300 tasks, we find that task success, privacy-compliant task completion, and later-session use of saved preferences are distinct capabilities.
  arXiv  Detail & Related papers  (2026-04-01T14:50:50Z)
• Anonymization-Enhanced Privacy Protection for Mobile GUI Agents: Available but Invisible [12.742325129012576]
  Mobile Graphical User Interface (GUI) agents have demonstrated strong capabilities in automating complex smartphone tasks.<n>We propose anonymization-based privacy protection framework that enforces the principle of available-but-invisible access to sensitive data.<n>Our system detects sensitive UI content using a PII-aware recognition model and replaces it with deterministic, type-preserving placeholders.
  arXiv  Detail & Related papers  (2026-02-08T15:50:04Z)
• Hijacking JARVIS: Benchmarking Mobile GUI Agents against Unprivileged Third Parties [19.430061128447022]
  We present the first systematic investigation into the vulnerabilities of mobile GUI agents.<n>We introduce a scalable attack simulation framework AgentHazard, which enables flexible and targeted modifications of screen content.<n>We develop a benchmark suite comprising both a dynamic task execution environment and a static dataset of attack scenarios.<n>Our findings reveal that all examined agents are significantly influenced by misleading third-party content.
  arXiv  Detail & Related papers  (2025-07-06T03:31:36Z)
• MAGPIE: A dataset for Multi-AGent contextual PrIvacy Evaluation [54.410825977390274]
  Existing benchmarks to evaluate contextual privacy in LLM-agents primarily assess single-turn, low-complexity tasks.<n>We first present a benchmark - MAGPIE comprising 158 real-life high-stakes scenarios across 15 domains.<n>We then evaluate the current state-of-the-art LLMs on their understanding of contextually private data and their ability to collaborate without violating user privacy.
  arXiv  Detail & Related papers  (2025-06-25T18:04:25Z)
• Poison Once, Control Anywhere: Clean-Text Visual Backdoors in VLM-based Mobile Agents [54.35629963816521]
  This work introduces VIBMA, the first clean-text backdoor attack targeting VLM-based mobile agents.<n>The attack injects malicious behaviors into the model by modifying only the visual input.<n>We show that our attack achieves high success rates while preserving clean-task behavior.
  arXiv  Detail & Related papers  (2025-06-16T08:09:32Z)
• AgentDAM: Privacy Leakage Evaluation for Autonomous Web Agents [66.29263282311258]
  We introduce a new benchmark AgentDAM that measures if AI web-navigation agents follow the privacy principle of data minimization''<n>Our benchmark simulates realistic web interaction scenarios end-to-end and is adaptable to all existing web navigation agents.
  arXiv  Detail & Related papers  (2025-03-12T19:30:31Z)
• {A New Hope}: Contextual Privacy Policies for Mobile Applications and An Approach Toward Automated Generation [19.578130824867596]
  The aim of contextual privacy policies ( CPPs) is to fragment privacy policies into concise snippets, displaying them only within the corresponding contexts within the application's graphical user interfaces (GUIs) In this paper, we first formulate CPP in mobile application scenario, and then present a novel multimodal framework, named SeePrivacy, specifically designed to automatically generate CPPs for mobile applications. A human evaluation shows that 77% of the extracted privacy policy segments were perceived as well-aligned with the detected contexts.
  arXiv  Detail & Related papers  (2024-02-22T13:32:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.