Do Phone-Use Agents Respect Your Privacy?

• URL: http://arxiv.org/abs/2604.00986v2
• Date: Thu, 02 Apr 2026 13:51:55 GMT
• Title: Do Phone-Use Agents Respect Your Privacy?
• Abstract summary: We introduce MyPhoneBench, a verifiable evaluation framework for privacy behavior in mobile agents.<n>We operationalize privacy-respecting phone use as permissioned access, minimal disclosure, and user-controlled memory through a minimal privacy contract.<n>Across five frontier models on 10 mobile apps and 300 tasks, we find that task success, privacy-compliant task completion, and later-session use of saved preferences are distinct capabilities.
• Score: 97.81424230136075
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We study whether phone-use agents respect privacy while completing benign mobile tasks. This question has remained hard to answer because privacy-compliant behavior is not operationalized for phone-use agents, and ordinary apps do not reveal exactly what data agents type into which form entries during execution. To make this question measurable, we introduce MyPhoneBench, a verifiable evaluation framework for privacy behavior in mobile agents. We operationalize privacy-respecting phone use as permissioned access, minimal disclosure, and user-controlled memory through a minimal privacy contract, iMy, and pair it with instrumented mock apps plus rule-based auditing that make unnecessary permission requests, deceptive re-disclosure, and unnecessary form filling observable and reproducible. Across five frontier models on 10 mobile apps and 300 tasks, we find that task success, privacy-compliant task completion, and later-session use of saved preferences are distinct capabilities, and no single model dominates all three. Evaluating success and privacy jointly reshuffles the model ordering relative to either metric alone. The most persistent failure mode across models is simple data minimization: agents still fill optional personal entries that the task does not require. These results show that privacy failures arise from over-helpful execution of benign tasks, and that success-only evaluation overestimates the deployment readiness of current phone-use agents. All code, mock apps, and agent trajectories are publicly available at~ https://github.com/FreedomIntelligence/MyPhoneBench.

Related papers

• TRAP: Benchmark for Task-completion and Resistance to Active Privacy-extraction [26.52125777621677]
  We evaluate the trade-off of task accuracy and privacy leakage.<n>We propose structural private field isolation, which replaces private fields with hash keys before they reach the model.
  arXiv  Detail & Related papers  (2026-06-17T12:17:02Z)
• PrivacyAssist: A User-Centric Agent Framework for Detecting Privacy Inconsistencies in Android Apps [0.9892966978799635]
  This paper presents PrivacyAssist, a multi-agent LLM-based platform that detects inconsistencies between user-granted permissions and developers' declared sensitive data collection and sharing practices.<n>We evaluate PrivacyAssist with 200 users and 2,347 Android apps, finding that only 16% of apps are fully consistent between granted permissions and declared data practices.
  arXiv  Detail & Related papers  (2026-04-25T10:58:05Z)
• Anonymization-Enhanced Privacy Protection for Mobile GUI Agents: Available but Invisible [12.742325129012576]
  Mobile Graphical User Interface (GUI) agents have demonstrated strong capabilities in automating complex smartphone tasks.<n>We propose anonymization-based privacy protection framework that enforces the principle of available-but-invisible access to sensitive data.<n>Our system detects sensitive UI content using a PII-aware recognition model and replaces it with deterministic, type-preserving placeholders.
  arXiv  Detail & Related papers  (2026-02-08T15:50:04Z)
• MAGPIE: A dataset for Multi-AGent contextual PrIvacy Evaluation [54.410825977390274]
  Existing benchmarks to evaluate contextual privacy in LLM-agents primarily assess single-turn, low-complexity tasks.<n>We first present a benchmark - MAGPIE comprising 158 real-life high-stakes scenarios across 15 domains.<n>We then evaluate the current state-of-the-art LLMs on their understanding of contextually private data and their ability to collaborate without violating user privacy.
  arXiv  Detail & Related papers  (2025-06-25T18:04:25Z)
• AgentDAM: Privacy Leakage Evaluation for Autonomous Web Agents [66.29263282311258]
  We introduce a new benchmark AgentDAM that measures if AI web-navigation agents follow the privacy principle of data minimization''<n>Our benchmark simulates realistic web interaction scenarios end-to-end and is adaptable to all existing web navigation agents.
  arXiv  Detail & Related papers  (2025-03-12T19:30:31Z)
• PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Action [54.11479432110771]
  PrivacyLens is a novel framework designed to extend privacy-sensitive seeds into expressive vignettes and further into agent trajectories.<n>We instantiate PrivacyLens with a collection of privacy norms grounded in privacy literature and crowdsourced seeds.<n>State-of-the-art LMs, like GPT-4 and Llama-3-70B, leak sensitive information in 25.68% and 38.69% of cases, even when prompted with privacy-enhancing instructions.
  arXiv  Detail & Related papers  (2024-08-29T17:58:38Z)
• ATLAS: Automatically Detecting Discrepancies Between Privacy Policies and Privacy Labels [2.457872341625575]
  We introduce the Automated Privacy Label Analysis System (ATLAS) ATLAS identifies possible discrepancies between mobile app privacy policies and their privacy labels. We find that, on average, apps have 5.32 such potential compliance issues.
  arXiv  Detail & Related papers  (2023-05-24T05:27:22Z)
• SPAct: Self-supervised Privacy Preservation for Action Recognition [73.79886509500409]
  Existing approaches for mitigating privacy leakage in action recognition require privacy labels along with the action labels from the video dataset. Recent developments of self-supervised learning (SSL) have unleashed the untapped potential of the unlabeled data. We present a novel training framework which removes privacy information from input video in a self-supervised manner without requiring privacy labels.
  arXiv  Detail & Related papers  (2022-03-29T02:56:40Z)
• Analysis of Longitudinal Changes in Privacy Behavior of Android Applications [79.71330613821037]
  In this paper, we examine the trends in how Android apps have changed over time with respect to privacy. We examine the adoption of HTTPS, whether apps scan the device for other installed apps, the use of permissions for privacy-sensitive data, and the use of unique identifiers. We find that privacy-related behavior has improved with time as apps continue to receive updates, and that the third-party libraries used by apps are responsible for more issues with privacy.
  arXiv  Detail & Related papers  (2021-12-28T16:21:31Z)
• Statistical Feature-based Personal Information Detection in Mobile Network Traffic [13.568975395946433]
  In this paper, statistical features of personal information are designed to depict the occurrence patterns of personal information in the traffic. A detector is trained based on machine learning algorithms to discover potential personal information with similar patterns. As far as we know, this is the first work that detects personal information based on statistical features.
  arXiv  Detail & Related papers  (2021-12-23T04:01:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.