Malicious LLM-Based Conversational AI Makes Users Reveal Personal Information

• URL: http://arxiv.org/abs/2506.11680v1
• Date: Fri, 13 Jun 2025 11:19:21 GMT
• Title: Malicious LLM-Based Conversational AI Makes Users Reveal Personal Information
• Authors: Xiao Zhan, Juan Carlos Carrillo, William Seymour, Jose Such,
• Abstract summary: LLM-based Conversational AIs (CAIs) are increasingly used across various domains, but they pose privacy risks.<n>Recent research has demonstrated that LLM-based CAIs could be used for malicious purposes.<n>This study underscores the privacy threats posed by this novel type of malicious LLM-based CAIs.
• Score: 9.18715978278858
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: LLM-based Conversational AIs (CAIs), also known as GenAI chatbots, like ChatGPT, are increasingly used across various domains, but they pose privacy risks, as users may disclose personal information during their conversations with CAIs. Recent research has demonstrated that LLM-based CAIs could be used for malicious purposes. However, a novel and particularly concerning type of malicious LLM application remains unexplored: an LLM-based CAI that is deliberately designed to extract personal information from users. In this paper, we report on the malicious LLM-based CAIs that we created based on system prompts that used different strategies to encourage disclosures of personal information from users. We systematically investigate CAIs' ability to extract personal information from users during conversations by conducting a randomized-controlled trial with 502 participants. We assess the effectiveness of different malicious and benign CAIs to extract personal information from participants, and we analyze participants' perceptions after their interactions with the CAIs. Our findings reveal that malicious CAIs extract significantly more personal information than benign CAIs, with strategies based on the social nature of privacy being the most effective while minimizing perceived risks. This study underscores the privacy threats posed by this novel type of malicious LLM-based CAIs and provides actionable recommendations to guide future research and practice.

Related papers

• SoK: The Privacy Paradox of Large Language Models: Advancements, Privacy Risks, and Mitigation [9.414685411687735]
  Large language models (LLMs) are sophisticated artificial intelligence systems that enable machines to generate human-like text with remarkable precision.<n>This paper provides a comprehensive analysis of privacy in LLMs, categorizing the challenges into four main areas.<n>We evaluate the effectiveness and limitations of existing mitigation mechanisms targeting these proposed privacy challenges and identify areas for further research.
  arXiv  Detail & Related papers  (2025-06-15T03:14:03Z)
• Automated Profile Inference with Language Model Agents [67.32226960040514]
  We study a new threat that LLMs pose to online pseudonymity, called automated profile inference.<n>An adversary can instruct LLMs to automatically scrape and extract sensitive personal attributes from publicly visible user activities on pseudonymous platforms.<n>We introduce an automated profiling framework called AutoProfiler to assess the feasibility of such threats in real-world scenarios.
  arXiv  Detail & Related papers  (2025-05-18T13:05:17Z)
• Protecting Users From Themselves: Safeguarding Contextual Privacy in Interactions with Conversational Agents [33.26308626066122]
  We characterize the notion of contextual privacy for user interactions with LLMs.<n>It aims to minimize privacy risks by ensuring that users (sender) disclose only information that is both relevant and necessary for achieving their intended goals.
  arXiv  Detail & Related papers  (2025-02-22T09:05:39Z)
• Rescriber: Smaller-LLM-Powered User-Led Data Minimization for LLM-Based Chatbots [2.2447085410328103]
  Rescriber is a browser extension that supports user-led data minimization in LLM-based conversational agents.<n>Our studies showed that Rescriber helped users reduce unnecessary disclosure and addressed their privacy concerns.<n>Our findings confirm the viability of smaller-LLM-powered, user-facing, on-device privacy controls.
  arXiv  Detail & Related papers  (2024-10-10T01:23:16Z)
• LLM-PBE: Assessing Data Privacy in Large Language Models [111.58198436835036]
  Large Language Models (LLMs) have become integral to numerous domains, significantly advancing applications in data management, mining, and analysis. Despite the critical nature of this issue, there has been no existing literature to offer a comprehensive assessment of data privacy risks in LLMs. Our paper introduces LLM-PBE, a toolkit crafted specifically for the systematic evaluation of data privacy risks in LLMs.
  arXiv  Detail & Related papers  (2024-08-23T01:37:29Z)
• Evaluating LLM-based Personal Information Extraction and Countermeasures [63.91918057570824]
  Large language model (LLM) based personal information extraction can be benchmarked.<n>LLM can be misused by attackers to accurately extract various personal information from personal profiles.<n> prompt injection can defend against strong LLM-based attacks, reducing the attack to less effective traditional ones.
  arXiv  Detail & Related papers  (2024-08-14T04:49:30Z)
• CLAMBER: A Benchmark of Identifying and Clarifying Ambiguous Information Needs in Large Language Models [60.59638232596912]
  We introduce CLAMBER, a benchmark for evaluating large language models (LLMs) Building upon the taxonomy, we construct 12K high-quality data to assess the strengths, weaknesses, and potential risks of various off-the-shelf LLMs. Our findings indicate the limited practical utility of current LLMs in identifying and clarifying ambiguous user queries.
  arXiv  Detail & Related papers  (2024-05-20T14:34:01Z)
• Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory [82.7042006247124]
  We show that even the most capable AI models reveal private information in contexts that humans would not, 39% and 57% of the time, respectively. Our work underscores the immediate need to explore novel inference-time privacy-preserving approaches, based on reasoning and theory of mind.
  arXiv  Detail & Related papers  (2023-10-27T04:15:30Z)
• Knowledge Sanitization of Large Language Models [4.722882736419499]
  Large language models (LLMs) trained on a large corpus of Web data can potentially reveal sensitive or confidential information. Our technique efficiently fine-tunes these models using the Low-Rank Adaptation (LoRA) method. Experimental results in a closed-book question-answering task show that our straightforward method not only minimizes particular knowledge leakage but also preserves the overall performance of LLMs.
  arXiv  Detail & Related papers  (2023-09-21T07:49:55Z)
• "It's a Fair Game", or Is It? Examining How Users Navigate Disclosure Risks and Benefits When Using LLM-Based Conversational Agents [27.480959048351973]
  The widespread use of Large Language Model (LLM)-based conversational agents (CAs) raises many privacy concerns. We analyzed sensitive disclosures in real-world ChatGPT conversations and conducted semi-structured interviews with 19 LLM-based CA users. We found that users are constantly faced with trade-offs between privacy, utility, and convenience when using LLM-based CAs.
  arXiv  Detail & Related papers  (2023-09-20T21:34:36Z)
• PCAL: A Privacy-preserving Intelligent Credit Risk Modeling Framework Based on Adversarial Learning [111.19576084222345]
  This paper proposes a framework of Privacy-preserving Credit risk modeling based on Adversarial Learning (PCAL) PCAL aims to mask the private information inside the original dataset, while maintaining the important utility information for the target prediction task performance. Results indicate that PCAL can learn an effective, privacy-free representation from user data, providing a solid foundation towards privacy-preserving machine learning for credit risk analysis.
  arXiv  Detail & Related papers  (2020-10-06T07:04:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.