PrivacySIM: Evaluating LLM Simulation of User Privacy Behavior

• URL: http://arxiv.org/abs/2605.12147v1
• Date: Tue, 12 May 2026 14:05:14 GMT
• Title: PrivacySIM: Evaluating LLM Simulation of User Privacy Behavior
• Abstract summary: Large language models (LLMs) are increasingly used to simulate human behavior, but their ability to simulate $individual$ privacy decisions is not well understood.<n>We introduce PrivacySIM, an evaluation suite that benchmarks LLM simulation of user privacy behavior against the ground-truth responses of 1,000 users.<n>We condition nine frontier LLMs on subsets of these three facets and measure how often each model's response to a data-sharing scenario matches the user's actual response.
• Score: 14.647995736798698
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) are increasingly used to simulate human behavior, but their ability to simulate $individual$ privacy decisions is not well understood. In this paper, we address the problem of evaluating whether a core set of user persona attributes can drive LLMs to simulate individual-level privacy behavior. We introduce PrivacySIM, an evaluation suite that benchmarks LLM simulation of user privacy behavior against the ground-truth responses of 1,000 users. These users are drawn from five published user studies on privacy spanning LLM healthcare consultations, conversational agents, and chatbots. Drawing on these user studies, we hypothesize three persona facets as plausible predictors of privacy decision-making: demographics, previous experiences, and stated privacy attitudes. We condition nine frontier LLMs on subsets of these three facets and measure how often each model's response to a data-sharing scenario matches the user's actual response. Our findings show that (1) privacy persona conditioning consistently improves simulation quality over no-persona conditioning, but even the strongest model (40.4\% accuracy) remains far from faithfully simulating individual privacy decisions. (2) A user's stated privacy attitudes alone may not be the best predictor because they often diverge from the user's actual privacy behavior. (3) Users with high AI/chatbot experience but low stated privacy attitudes are the most challenging to simulate. PrivacySIM is a first step toward understanding and improving the capabilities of LLMs to simulate user privacy decisions. We release PrivacySIM to enable further evaluation of LLM privacy simulation.

Related papers

• Seeing Without Exposing: Adaptive Privacy Control for Open-World, Context-Hungry MLLMs [61.80513991207956]
  We propose Anchored Privacy Drifting (APD) to tackle privacy challenges in MLLMs.<n>APD drifts privacy-sensitive elements toward semantically equivalent alternatives while anchoring contextual cues to the source image.<n>We show that our method achieves balanced improvements in both privacy sanitization and content retention.
  arXiv  Detail & Related papers  (2026-06-05T11:41:39Z)
• Profiling for Pennies: Unveiling the Privacy Iceberg of LLM Agents [52.02752011119451]
  We investigate privacy perception in the real world through the lens of human awareness.<n>We propose the PrivacyIceberg, which categorizes real-world human privacy risks into three tiers.
  arXiv  Detail & Related papers  (2026-05-07T13:21:44Z)
• Text-Based Personas for Simulating User Privacy Decisions [23.902474542548944]
  We present Narriva, an approach that generates text-based synthetic privacy personas.<n>We analyze the characteristics of Narriva's synthetic personas in modeling both individual and population-level privacy preferences.
  arXiv  Detail & Related papers  (2026-03-20T09:28:15Z)
• VoxPrivacy: A Benchmark for Evaluating Interactional Privacy of Speech Language Models [25.266028200777317]
  Speech Language Models (SLMs) are expected to distinguish between users to manage information flow appropriately.<n>Current SLM benchmarks test dialogue ability but overlook speaker identity.<n>We introduce VoxPrivacy, the first benchmark designed to evaluate interactional privacy in SLMs.
  arXiv  Detail & Related papers  (2026-01-27T06:22:14Z)
• PrivacyReasoner: Can LLM Emulate a Human-like Privacy Mind? [13.499949825312797]
  This paper introduces PRA, an AI-agent design for simulating how individual users form privacy concerns in response to real-world news.<n>Experiments on real-world Hacker News discussions show that PRA outperforms baseline agents in privacy concern prediction.
  arXiv  Detail & Related papers  (2026-01-14T04:47:06Z)
• User Perceptions of Privacy and Helpfulness in LLM Responses to Privacy-Sensitive Scenarios [10.12906605142667]
  We show how users perceive privacy-preservation quality and helpfulness of large language models responses to privacy-sensitive scenarios.<n>Our results suggest the need to conduct user-centered studies on measuring LLMs' ability to help users while preserving privacy.
  arXiv  Detail & Related papers  (2025-10-23T16:38:26Z)
• LLM-as-a-Judge for Privacy Evaluation? Exploring the Alignment of Human and LLM Perceptions of Privacy in Textual Data [47.76073133338117]
  Despite advances in the field of privacy- Natural Language Processing (NLP), the accurate evaluation of privacy remains a challenge.<n>We present a global approach inspired by sox2013$ a model for privacy evaluation in textual data.<n>Our findings pave the way for exploring the feasibility of evaluators as privacy evaluators.
  arXiv  Detail & Related papers  (2025-08-16T20:49:41Z)
• Benchmarking LLM Privacy Recognition for Social Robot Decision Making [21.331666364341448]
  We present a set of privacy-relevant scenarios developed using the Contextual Integrity (CI) framework.<n>We found that the agreement between humans and LLMs was generally low.<n>We discuss the performance of the evaluated models as well as the implications and potential of AI privacy awareness in human-robot interaction.
  arXiv  Detail & Related papers  (2025-07-22T00:36:59Z)
• MAGPIE: A dataset for Multi-AGent contextual PrIvacy Evaluation [54.410825977390274]
  Existing benchmarks to evaluate contextual privacy in LLM-agents primarily assess single-turn, low-complexity tasks.<n>We first present a benchmark - MAGPIE comprising 158 real-life high-stakes scenarios across 15 domains.<n>We then evaluate the current state-of-the-art LLMs on their understanding of contextually private data and their ability to collaborate without violating user privacy.
  arXiv  Detail & Related papers  (2025-06-25T18:04:25Z)
• PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Action [54.11479432110771]
  PrivacyLens is a novel framework designed to extend privacy-sensitive seeds into expressive vignettes and further into agent trajectories.<n>We instantiate PrivacyLens with a collection of privacy norms grounded in privacy literature and crowdsourced seeds.<n>State-of-the-art LMs, like GPT-4 and Llama-3-70B, leak sensitive information in 25.68% and 38.69% of cases, even when prompted with privacy-enhancing instructions.
  arXiv  Detail & Related papers  (2024-08-29T17:58:38Z)
• Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory [82.7042006247124]
  We show that even the most capable AI models reveal private information in contexts that humans would not, 39% and 57% of the time, respectively. Our work underscores the immediate need to explore novel inference-time privacy-preserving approaches, based on reasoning and theory of mind.
  arXiv  Detail & Related papers  (2023-10-27T04:15:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.