When Are LLM Inferences Acceptable? User Reactions and Control Preferences for Inferred Personal Information

• URL: http://arxiv.org/abs/2605.10013v1
• Date: Mon, 11 May 2026 05:39:48 GMT
• Title: When Are LLM Inferences Acceptable? User Reactions and Control Preferences for Inferred Personal Information
• Abstract summary: We built a visualization tool that surfaces example unstated inferences from users' own ChatGPT histories.<n>Discomfort arose mainly when inferences felt misrepresentative of the user or misaligned with expected use.<n>These findings suggest that the acceptability of LLM inferences is governed not only by its content, but by context-sensitive norms around how they are generated.
• Score: 44.18115610488797
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Ask ChatGPT about vacation planning, and it may infer your income. Ask it about medication, and it may infer your medical history. Because such inferences can expose more information than users intend to reveal, prior work argues that they are a defining privacy risk of LLM-based systems. Yet prior work has mostly shown that LLMs can make potentially violating inferences, not how users experience those inferences nor what controls users may want governing their use. We built the Reflective Layer, a visualization tool that surfaces example unstated inferences from users' own ChatGPT histories, and used it in a mixed-methods study with 18 regular ChatGPT users evaluating 215 surfaced inferences from their own conversations. Counterintuitively, participants reacted more strongly with curiosity and interest rather than distress and concern. Discomfort arose mainly when inferences felt misrepresentative of the user or misaligned with expected use. Participants were also markedly less comfortable with advertisers and third-party applications using those inferences than with platform providers. These findings suggest that the acceptability of LLM inferences is governed not only by its content, but by context-sensitive norms around how they are generated, retained within the platform, and transmitted beyond it.

Related papers

• Verbalizing LLMs' assumptions to explain and control sycophancy [62.927670321859495]
  LLMs can be socially sycophantic, affirming users when they ask questions like "am I in the wrong?" rather than providing genuine assessment.<n>We present Verbalized Assumptions, a framework for eliciting these assumptions from LLMs.<n>Our work contributes a new understanding of assumptions as a mechanism for sycophancy.
  arXiv  Detail & Related papers  (2026-04-03T14:15:43Z)
• Towards Realistic Personalization: Evaluating Long-Horizon Preference Following in Personalized User-LLM Interactions [50.70965714314064]
  Large Language Models (LLMs) are increasingly serving as personal assistants, where users share complex and diverse preferences over extended interactions.<n>This work proposes RealPref, a benchmark for evaluating realistic preference-following in personalized user-LLM interactions.
  arXiv  Detail & Related papers  (2026-03-04T15:42:43Z)
• Exploring User Acceptance and Concerns toward LLM-powered Conversational Agents in Immersive Extended Reality [16.53846784748676]
  The extended reality (XR) community has sought to integrate large language models (LLMs) to enhance user experience and task efficiency.<n>While users generally accept these technologies, they express concerns related to security, privacy, social implications, and trust.<n>Our results suggest that familiarity plays a crucial role, as daily generative AI use is associated with greater acceptance.
  arXiv  Detail & Related papers  (2025-12-17T11:41:25Z)
• User Perceptions of Privacy and Helpfulness in LLM Responses to Privacy-Sensitive Scenarios [10.12906605142667]
  We show how users perceive privacy-preservation quality and helpfulness of large language models responses to privacy-sensitive scenarios.<n>Our results suggest the need to conduct user-centered studies on measuring LLMs' ability to help users while preserving privacy.
  arXiv  Detail & Related papers  (2025-10-23T16:38:26Z)
• Beyond PII: How Users Attempt to Estimate and Mitigate Implicit LLM Inference [8.063685458567202]
  Large Language Models (LLMs) can infer personal attributes from seemingly innocuous text, raising privacy risks beyond memorized data leakage.<n>We conducted a survey with 240 U.S. participants who judged text snippets for inference risks, reported concern levels, and attempted rewrites to block inference.<n>Results show that participants struggled to anticipate inference, performing a little better than chance.
  arXiv  Detail & Related papers  (2025-09-15T17:17:26Z)
• Controlling What You Share: Assessing Language Model Adherence to Privacy Preferences [73.5779077857545]
  We build a framework where a local model uses these instructions to rewrite queries, only hiding details deemed sensitive by the user, before sending them to an external model.<n>Experiments with lightweight local LLMs show that, after fine-tuning, they markedly exceed the performance of much larger zero-shot models.<n>At the same time, the system still faces challenges in fully adhering to user instructions, underscoring the need for models with a better understanding of user-defined privacy preferences.
  arXiv  Detail & Related papers  (2025-07-07T18:22:55Z)
• Rescriber: Smaller-LLM-Powered User-Led Data Minimization for LLM-Based Chatbots [2.2447085410328103]
  Rescriber is a browser extension that supports user-led data minimization in LLM-based conversational agents.<n>Our studies showed that Rescriber helped users reduce unnecessary disclosure and addressed their privacy concerns.<n>Our findings confirm the viability of smaller-LLM-powered, user-facing, on-device privacy controls.
  arXiv  Detail & Related papers  (2024-10-10T01:23:16Z)
• Prompt Tuning as User Inherent Profile Inference Machine [68.16976932088708]
  We propose UserIP-Tuning, which uses prompt-tuning to infer user profiles.<n>UserIP-Tuning outperforms state-of-the-art recommendation algorithms.<n>The presented solution has been deployed in Huawei AppGallery's Explore page since May 2025.
  arXiv  Detail & Related papers  (2024-08-13T02:25:46Z)
• "I'm Not Sure, But...": Examining the Impact of Large Language Models' Uncertainty Expression on User Reliance and Trust [51.542856739181474]
  We show how different natural language expressions of uncertainty impact participants' reliance, trust, and overall task performance. We find that first-person expressions decrease participants' confidence in the system and tendency to agree with the system's answers, while increasing participants' accuracy. Our findings suggest that using natural language expressions of uncertainty may be an effective approach for reducing overreliance on LLMs, but that the precise language used matters.
  arXiv  Detail & Related papers  (2024-05-01T16:43:55Z)
• "It's a Fair Game", or Is It? Examining How Users Navigate Disclosure Risks and Benefits When Using LLM-Based Conversational Agents [27.480959048351973]
  The widespread use of Large Language Model (LLM)-based conversational agents (CAs) raises many privacy concerns. We analyzed sensitive disclosures in real-world ChatGPT conversations and conducted semi-structured interviews with 19 LLM-based CA users. We found that users are constantly faced with trade-offs between privacy, utility, and convenience when using LLM-based CAs.
  arXiv  Detail & Related papers  (2023-09-20T21:34:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.