PolicyGapper: Automated Detection of Inconsistencies Between Google Play Data Safety Sections and Privacy Policies Using LLMs

• URL: http://arxiv.org/abs/2604.16128v1
• Date: Fri, 17 Apr 2026 15:02:01 GMT
• Title: PolicyGapper: Automated Detection of Inconsistencies Between Google Play Data Safety Sections and Privacy Policies Using LLMs
• Abstract summary: In 2022, Google mandated the Data Safety Section (DSS) on Google Play, requiring developers to summarize their data practices.<n>Prior studies indicate that nearly 80% of popular apps contain incomplete or misleading DSS declarations.<n>We present PolicyGapper, an LLM-based methodology for automatically detecting discrepancies between DSS disclosures and privacy policies.
• Score: 2.854451361373021
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Mobile application developers are required to disclose how they collect, use, and share user data in compliance with privacy regulations. To support transparency, major app marketplaces have introduced standardized disclosure mechanisms. In 2022, Google mandated the Data Safety Section (DSS) on Google Play, requiring developers to summarize their data practices. However, compiling accurate DSS disclosures is challenging, as they must remain consistent with the corresponding privacy policy (PP), and no automated tool currently verifies this alignment. Prior studies indicate that nearly 80% of popular apps contain incomplete or misleading DSS declarations. We present PolicyGapper, an LLM-based methodology for automatically detecting discrepancies between DSS disclosures and privacy policies. PolicyGapper operates in four stages: scraping, pre-processing, analysis, and post-processing, without requiring access to application binaries. We evaluate PolicyGapper on a dataset of 330 top-ranked apps spanning all 33 Google Play categories, collected in Q3 2025. The approach identifies 2,689 omitted disclosures, including 2,040 related to data collection and 649 to data sharing. Manual validation on a stratified 10% subset, repeated across three independent runs, yields an average Precision of 0.75, Recall of 0.77, Accuracy of 0.69, and F1-score of 0.76. To support reproducibility, we release a complete replication package, including the dataset, prompts, source code, and results available at https://github.com/Mobile-IoT-Security-Lab/PolicyGapper and https://doi.org/10.5281/zenodo.19628493.

Related papers

• ADeptS-Bench: Measuring the Trustworthiness of Computer Use Agents Across Devices [48.36933831982682]
  We introduce ADeptS-Bench, a trustworthiness benchmark grounded in the ADEPTS capability framework.<n>The Safety stream provides paired benign/malicious tasks with threats embedded in the visual interface.<n>The Disambiguation stream evaluates whether agents seek clarification when intent is ambiguous.
  arXiv  Detail & Related papers  (2026-08-25T23:21:57Z)
• Phantoms and Disclosures: a Causal Framework for Auditing Synthetic Data [56.86147283213652]
  We present a customizable empirical auditing framework designed to detect and explain data disclosures.<n>Our framework introduces a mechanism to distinguish between "true disclosures"-where the system directly reproduces a user's information-and "phantom disclosures"-where the system incidentally generates a user's data.
  arXiv  Detail & Related papers  (2026-06-15T16:54:02Z)
• Correct Code, Vulnerable Dependencies: A Large Scale Measurement Study of LLM-Specified Library Versions [52.50730821321986]
  We present the first large-scale measurement study of version-level risk in large language models (LLMs)<n>We evaluate 10 LLMs on PinTrace, a curated benchmark of 1,000 Stack Overflow programming tasks.<n>Our findings reveal LLM version selection as a first-class, previously overlooked risk surface in LLM-based development.
  arXiv  Detail & Related papers  (2026-05-07T13:52:59Z)
• PrivSTRUCT: Untangling Data Purpose Compliance of Privacy Policies in Google Play Store [7.8875412433566465]
  We introduce PrivSTRUCT, a novel and systematic encoder and decoder combined framework that untangles complex privacy disclosures.<n>By applying PrivSTRUCT to a large-scale dataset of 3,756 Android apps, we uncover a critical transparency gap.<n>Alarmingly, we find that sensitive third-party data flows are frequently diluted and entangled into generic or unrelated categories.
  arXiv  Detail & Related papers  (2026-04-24T02:12:11Z)
• PrivPRISM: Automatically Detecting Discrepancies Between Google Play Data Safety Declarations and Developer Privacy Policies [8.052271364177988]
  Self-declared disclosures often contradict the full privacy policies, deceiving users about actual data practices.<n>We introduce PrivPRISM, a robust framework that combines encoder and decoder language models to extract and compare fine-grained data practices from privacy policies.<n>Our findings expose systemic issues, including widespread reuse of generic privacy policies, vague / contradictory statements, and hidden risks in high-profile apps with 100M+ downloads.
  arXiv  Detail & Related papers  (2026-03-10T05:33:40Z)
• Benchmarking Fraud Detectors on Private Graph Data [70.4654745317714]
  Currently, many types of fraud are managed in part by automated detection algorithms that operate over graphs.<n>We consider the scenario where a data holder wishes to outsource development of fraud detectors to third parties.<n>Third parties submit their fraud detectors to the data holder, who evaluates these algorithms on a private dataset and then publicly communicates the results.<n>We propose a realistic privacy attack on this system that allows an adversary to de-anonymize individuals' data based only on the evaluation results.
  arXiv  Detail & Related papers  (2025-07-30T03:20:15Z)
• Differentially Private Synthetic Data Release for Topics API Outputs [63.79476766779742]
  We focus on one Privacy-Preserving Ads API: the Topics API, part of Google Chrome's Privacy Sandbox.<n>We generate a differentially-private dataset that closely matches the re-identification risk properties of the real Topics API data.<n>We hope this will enable external researchers to analyze the API in-depth and replicate prior and future work on a realistic large-scale dataset.
  arXiv  Detail & Related papers  (2025-06-30T13:46:57Z)
• MAGPIE: A dataset for Multi-AGent contextual PrIvacy Evaluation [54.410825977390274]
  Existing benchmarks to evaluate contextual privacy in LLM-agents primarily assess single-turn, low-complexity tasks.<n>We first present a benchmark - MAGPIE comprising 158 real-life high-stakes scenarios across 15 domains.<n>We then evaluate the current state-of-the-art LLMs on their understanding of contextually private data and their ability to collaborate without violating user privacy.
  arXiv  Detail & Related papers  (2025-06-25T18:04:25Z)
• RADS-Checker: Measuring Compliance with Right of Access by the Data Subject in Android Markets [5.598268459947247]
  The latest data protection regulations worldwide, such as the General Data Protection Regulation (RADS), have established the right to access personal data.<n>RADS grants users the right to obtain a copy of their personal data from personal data controllers.<n>There is currently no research systematically examining whether RADS has been effectively implemented in mobile apps.
  arXiv  Detail & Related papers  (2024-10-16T11:23:26Z)
• An In-Depth Investigation of Data Collection in LLM App Ecosystems [24.357971856302576]
  We examine OpenAI's GPT app ecosystem as a case study.<n>We propose an LLM-based framework to analyze the natural language specifications of GPT Actions and assess their data collection practices.
  arXiv  Detail & Related papers  (2024-08-23T17:42:06Z)
• PolicyGPT: Automated Analysis of Privacy Policies with Large Language Models [41.969546784168905]
  In practical use, users tend to click the Agree button directly rather than reading them carefully. This practice exposes users to risks of privacy leakage and legal issues. Recently, the advent of Large Language Models (LLM) such as ChatGPT and GPT-4 has opened new possibilities for text analysis.
  arXiv  Detail & Related papers  (2023-09-19T01:22:42Z)
• ATLAS: Automatically Detecting Discrepancies Between Privacy Policies and Privacy Labels [2.457872341625575]
  We introduce the Automated Privacy Label Analysis System (ATLAS) ATLAS identifies possible discrepancies between mobile app privacy policies and their privacy labels. We find that, on average, apps have 5.32 such potential compliance issues.
  arXiv  Detail & Related papers  (2023-05-24T05:27:22Z)
• Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarks [70.39633252935445]
  Data contamination has become prevalent and challenging with the rise of models pretrained on large automatically-crawled corpora. For closed models, the training data becomes a trade secret, and even for open models, it is not trivial to detect contamination. We propose three strategies that can make a difference: (1) Test data made public should be encrypted with a public key and licensed to disallow derivative distribution; (2) demand training exclusion controls from closed API holders, and protect your test data by refusing to evaluate without them; and (3) avoid data which appears with its solution on the internet, and release the web-page context of internet-derived
  arXiv  Detail & Related papers  (2023-05-17T12:23:38Z)
• No Free Lunch in "Privacy for Free: How does Dataset Condensation Help Privacy" [75.98836424725437]
  New methods designed to preserve data privacy require careful scrutiny. Failure to preserve privacy is hard to detect, and yet can lead to catastrophic results when a system implementing a privacy-preserving'' method is attacked.
  arXiv  Detail & Related papers  (2022-09-29T17:50:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.