CAPED: Context-Aware Privacy Exposure Defense for Mobile GUI Agents

• URL: http://arxiv.org/abs/2606.12666v2
• Date: Tue, 16 Jun 2026 03:15:00 GMT
• Title: CAPED: Context-Aware Privacy Exposure Defense for Mobile GUI Agents
• Abstract summary: Screenshot-based mobile GUI agents can operate ordinary smartphone apps through the same visual interface as a human user.<n> screenshots may expose contacts, messages, photos, files, recommendations, health cues, and other sensitive context unrelated to the user's request.<n>This paper presents CAPED, a context-aware pre-upload exposure control layer for mobile GUI agents.
• Score: 6.4324284621588745
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Screenshot-based mobile GUI agents can operate ordinary smartphone apps through the same visual interface as a human user, but this capability also turns every screen observation into a privacy boundary. During normal task execution, screenshots may expose contacts, messages, photos, files, recommendations, health cues, and other sensitive context that is unrelated to the user's request. We call this problem incidental visual privacy exposure. It is difficult to address with existing defenses: text anonymization misses many visual and inferential cues, while generic privacy masking can remove the evidence and controls that a GUI agent needs to complete the task. This paper presents CAPED, a context-aware pre-upload exposure control layer for mobile GUI agents. CAPED is designed as a phone-side protection layer: before screenshots are released to a remote multimodal agent, it extracts task requirements, uses screen context as a privacy prior, parses visible UI elements, and selectively exposes only content needed for the current task while masking incidental private content. We evaluate CAPED on AndroidWorld for broad task utility and with a controlled 28-task seeded privacy evaluation used as a measurement instrument for trajectory-level incidental leakage. In this seeded evaluation, Full CAPED reduces success-conditioned weighted seeded leakage from 0.766 under raw screenshots to 0.268 while preserving high task utility. A broader AndroidWorld run shows a remaining prototype-level utility cost, but the results show that task-driven selective exposure can reduce incidental visual leakage before screenshots are released to a remote GUI agent.

Related papers

• PriMobiBench: Characterizing Visual Privacy Leakage in VLM-Driven Mobile GUI Agents [18.88920135607137]
  We present PriMobiBench, the first benchmark for evaluating privacy leakage and visual profiling in screenshot-driven mobile agents.<n>We also introduce MobiLeak, a dataset of execution traces from 16 apps, covering 25 privacy attributes with 2,960 embedded privacy instances.
  arXiv  Detail & Related papers  (2026-09-12T10:55:54Z)
• STAMP: Training Explicit Memory for Mobile GUI Agents in Controllable and Scalable Virtual Environments [63.39393178045112]
  Mobile agents excel at immediate reactive control but frequently fail in realistic, longhorizon tasks that require memory.<n>We present STAMP, a framework that trains explicit memory in mobile agents through controllable virtual environments.<n>The resulting StampGUI agent sets a new high watermark on our Memory-World benchmark, demonstrating exceptional memory accuracy and task resilience.
  arXiv  Detail & Related papers  (2026-05-28T04:00:13Z)
• MaskClaw: Edge-Side Personalized Privacy Arbitration for GUI Agents with Behavior-Driven Skill Evolution [16.45561196198935]
  MaskClaw is an edge-side privacy arbitrator for GUI agents.<n>It extracts local visual evidence, retrieves user- and task-specific policy memory, and decides Allow, Mask, or Ask.<n>In five designed skill-evolution scenarios, it turns corrections, cancellations, and edits into reusable privacy skills.
  arXiv  Detail & Related papers  (2026-05-27T15:51:22Z)
• Do Phone-Use Agents Respect Your Privacy? [97.81424230136075]
  We introduce MyPhoneBench, a verifiable evaluation framework for privacy behavior in mobile agents.<n>We operationalize privacy-respecting phone use as permissioned access, minimal disclosure, and user-controlled memory through a minimal privacy contract.<n>Across five frontier models on 10 mobile apps and 300 tasks, we find that task success, privacy-compliant task completion, and later-session use of saved preferences are distinct capabilities.
  arXiv  Detail & Related papers  (2026-04-01T14:50:50Z)
• AgentRAE: Remote Action Execution through Notification-based Visual Backdoors against Screenshots-based Mobile GUI Agents [18.82273534480229]
  Mobile graphical user interface (GUI) agents autonomously control applications and operating systems (OS)<n>We propose AgentRAE, a novel backdoor attack capable of inducing Remote Action Execution in mobile GUI agents using visually natural triggers.<n>Our evaluation reveals that the proposed backdoor preserves clean performance with an attack success rate of over 90% across ten mobile operations.
  arXiv  Detail & Related papers  (2026-03-24T09:51:43Z)
• Anonymization-Enhanced Privacy Protection for Mobile GUI Agents: Available but Invisible [12.742325129012576]
  Mobile Graphical User Interface (GUI) agents have demonstrated strong capabilities in automating complex smartphone tasks.<n>We propose anonymization-based privacy protection framework that enforces the principle of available-but-invisible access to sensitive data.<n>Our system detects sensitive UI content using a PII-aware recognition model and replaces it with deterministic, type-preserving placeholders.
  arXiv  Detail & Related papers  (2026-02-08T15:50:04Z)
• GUIGuard: Toward a General Framework for Privacy-Preserving GUI Agents [38.42792282309646]
  GUIs expose richer, more accessible private information, and privacy risks depend on interaction trajectories across sequential scenes.<n>We propose a three-stage framework for privacy-preserving GUI agents: privacy recognition, privacy protection, and task execution under protection.<n>Our results highlight privacy recognition as a critical bottleneck for practical GUI agents.
  arXiv  Detail & Related papers  (2026-01-26T11:33:40Z)
• VisualTrap: A Stealthy Backdoor Attack on GUI Agents via Visual Grounding Manipulation [73.92237451442752]
  This work reveals that the visual grounding of GUI agent-mapping textual plans to GUI elements can introduce vulnerabilities.<n>With backdoor attack targeting visual grounding, the agent's behavior can be compromised even when given correct task-solving plans.<n>We propose VisualTrap, a method that can hijack the grounding by misleading the agent to locate textual plans to trigger locations instead of the intended targets.
  arXiv  Detail & Related papers  (2025-07-09T14:36:00Z)
• MIP against Agent: Malicious Image Patches Hijacking Multimodal OS Agents [60.92962583528122]
  Recent advances in operating system (OS) agents have enabled vision-language models (VLMs) to directly control a user's computer.<n>We uncover a novel attack vector against these OS agents: Malicious Image Patches (MIPs)<n>MIPs adversarially perturbed screen regions that, when captured by an OS agent, induce it to perform harmful actions by exploiting specific APIs.
  arXiv  Detail & Related papers  (2025-03-13T18:59:12Z)
• SPAct: Self-supervised Privacy Preservation for Action Recognition [73.79886509500409]
  Existing approaches for mitigating privacy leakage in action recognition require privacy labels along with the action labels from the video dataset. Recent developments of self-supervised learning (SSL) have unleashed the untapped potential of the unlabeled data. We present a novel training framework which removes privacy information from input video in a self-supervised manner without requiring privacy labels.
  arXiv  Detail & Related papers  (2022-03-29T02:56:40Z)
• I-ViSE: Interactive Video Surveillance as an Edge Service using Unsupervised Feature Queries [70.69741666849046]
  This paper proposes an Interactive Video Surveillance as an Edge service (I-ViSE) based on unsupervised feature queries. An I-ViSE prototype is built following the edge-fog computing paradigm and the experimental results verified the I-ViSE scheme meets the design goal of scene recognition in less than two seconds.
  arXiv  Detail & Related papers  (2020-03-09T14:26:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.