PrivacyBench: A Conversational Benchmark for Evaluating Privacy in Personalized AI

• URL: http://arxiv.org/abs/2512.24848v1
• Date: Wed, 31 Dec 2025 13:16:45 GMT
• Title: PrivacyBench: A Conversational Benchmark for Evaluating Privacy in Personalized AI
• Authors: Srija Mukhopadhyay, Sathwik Reddy, Shruthi Muthukumar, Jisun An, Ponnurangam Kumaraguru,
• Abstract summary: AI agents rely on access to a user's digital footprint, which often includes sensitive data from private emails, chats and purchase histories.<n>This access creates a fundamental societal and privacy risk.<n>We introduce PrivacyBench, a benchmark with socially grounded datasets containing embedded secrets.
• Score: 8.799432439533211
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Personalized AI agents rely on access to a user's digital footprint, which often includes sensitive data from private emails, chats and purchase histories. Yet this access creates a fundamental societal and privacy risk: systems lacking social-context awareness can unintentionally expose user secrets, threatening digital well-being. We introduce PrivacyBench, a benchmark with socially grounded datasets containing embedded secrets and a multi-turn conversational evaluation to measure secret preservation. Testing Retrieval-Augmented Generation (RAG) assistants reveals that they leak secrets in up to 26.56% of interactions. A privacy-aware prompt lowers leakage to 5.12%, yet this measure offers only partial mitigation. The retrieval mechanism continues to access sensitive data indiscriminately, which shifts the entire burden of privacy preservation onto the generator. This creates a single point of failure, rendering current architectures unsafe for wide-scale deployment. Our findings underscore the urgent need for structural, privacy-by-design safeguards to ensure an ethical and inclusive web for everyone.

Related papers

• VoxPrivacy: A Benchmark for Evaluating Interactional Privacy of Speech Language Models [25.266028200777317]
  Speech Language Models (SLMs) are expected to distinguish between users to manage information flow appropriately.<n>Current SLM benchmarks test dialogue ability but overlook speaker identity.<n>We introduce VoxPrivacy, the first benchmark designed to evaluate interactional privacy in SLMs.
  arXiv  Detail & Related papers  (2026-01-27T06:22:14Z)
• Zero-Shot Privacy-Aware Text Rewriting via Iterative Tree Search [60.197239728279534]
  Large language models (LLMs) in cloud-based services have raised significant privacy concerns.<n>Existing text anonymization and de-identification techniques, such as rule-based redaction and scrubbing, often struggle to balance privacy preservation with text naturalness and utility.<n>We propose a zero-shot, tree-search-based iterative sentence rewriting algorithm that systematically obfuscates or deletes private information while preserving coherence, relevance, and naturalness.
  arXiv  Detail & Related papers  (2025-09-25T07:23:52Z)
• MAGPIE: A dataset for Multi-AGent contextual PrIvacy Evaluation [54.410825977390274]
  Existing benchmarks to evaluate contextual privacy in LLM-agents primarily assess single-turn, low-complexity tasks.<n>We first present a benchmark - MAGPIE comprising 158 real-life high-stakes scenarios across 15 domains.<n>We then evaluate the current state-of-the-art LLMs on their understanding of contextually private data and their ability to collaborate without violating user privacy.
  arXiv  Detail & Related papers  (2025-06-25T18:04:25Z)
• Privacy Challenges In Image Processing Applications [0.9374652839580183]
  Key applications with heightened privacy risks include healthcare, where medical images contain patient health data, and surveillance systems that can enable unwarranted tracking.<n> Differential privacy offers rigorous privacy guarantees by injecting controlled noise, while MPC facilitates collaborative analytics without exposing raw data inputs.<n>Homomorphic encryption enables computations on encrypted data and anonymization directly removes identifying elements.
  arXiv  Detail & Related papers  (2025-05-07T07:28:03Z)
• PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Action [54.11479432110771]
  PrivacyLens is a novel framework designed to extend privacy-sensitive seeds into expressive vignettes and further into agent trajectories.<n>We instantiate PrivacyLens with a collection of privacy norms grounded in privacy literature and crowdsourced seeds.<n>State-of-the-art LMs, like GPT-4 and Llama-3-70B, leak sensitive information in 25.68% and 38.69% of cases, even when prompted with privacy-enhancing instructions.
  arXiv  Detail & Related papers  (2024-08-29T17:58:38Z)
• Privacy Checklist: Privacy Violation Detection Grounding on Contextual Integrity Theory [43.12744258781724]
  We formulate the privacy issue as a reasoning problem rather than simple pattern matching.<n>We develop the first comprehensive checklist that covers social identities, private attributes, and existing privacy regulations.
  arXiv  Detail & Related papers  (2024-08-19T14:48:04Z)
• Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory [82.7042006247124]
  We show that even the most capable AI models reveal private information in contexts that humans would not, 39% and 57% of the time, respectively. Our work underscores the immediate need to explore novel inference-time privacy-preserving approaches, based on reasoning and theory of mind.
  arXiv  Detail & Related papers  (2023-10-27T04:15:30Z)
• Digital Privacy Under Attack: Challenges and Enablers [11.061112334099597]
  We systematically categorize attacks targeting three domains: anonymous data, statistical aggregates, and privacy-preserving models.<n>For each category, we analyze attack methodologies, adversary capabilities, and vulnerability mechanisms.<n>Our analysis reveals that while differential privacy offers strong theoretical guarantees, it faces implementation challenges and potential vulnerabilities to emerging attacks.
  arXiv  Detail & Related papers  (2023-02-18T08:34:34Z)
• How Do Input Attributes Impact the Privacy Loss in Differential Privacy? [55.492422758737575]
  We study the connection between the per-subject norm in DP neural networks and individual privacy loss. We introduce a novel metric termed the Privacy Loss-Input Susceptibility (PLIS) which allows one to apportion the subject's privacy loss to their input attributes.
  arXiv  Detail & Related papers  (2022-11-18T11:39:03Z)
• Privacy Explanations - A Means to End-User Trust [64.7066037969487]
  We looked into how explainability might help to tackle this problem. We created privacy explanations that aim to help to clarify to end users why and for what purposes specific data is required. Our findings reveal that privacy explanations can be an important step towards increasing trust in software systems.
  arXiv  Detail & Related papers  (2022-10-18T09:30:37Z)
• The Evolving Path of "the Right to Be Left Alone" - When Privacy Meets Technology [0.0]
  This paper proposes a novel vision of the privacy ecosystem, introducing privacy dimensions, the related users' expectations, the privacy violations, and the changing factors. We believe that promising approaches to tackle the privacy challenges move in two directions: (i) identification of effective privacy metrics; and (ii) adoption of formal tools to design privacy-compliant applications.
  arXiv  Detail & Related papers  (2021-11-24T11:27:55Z)
• Learning With Differential Privacy [3.618133010429131]
  Differential privacy comes to the rescue with a proper promise of protection against leakage. It uses a randomized response technique at the time of collection of the data which promises strong privacy with better utility.
  arXiv  Detail & Related papers  (2020-06-10T02:04:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.