Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory

• URL: http://arxiv.org/abs/2310.17884v2
• Date: Fri, 28 Jun 2024 23:27:10 GMT
• Title: Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory
• Authors: Niloofar Mireshghallah, Hyunwoo Kim, Xuhui Zhou, Yulia Tsvetkov, Maarten Sap, Reza Shokri, Yejin Choi,
• Abstract summary: We show that even the most capable AI models reveal private information in contexts that humans would not, 39% and 57% of the time, respectively. Our work underscores the immediate need to explore novel inference-time privacy-preserving approaches, based on reasoning and theory of mind.
• Score: 82.7042006247124
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The interactive use of large language models (LLMs) in AI assistants (at work, home, etc.) introduces a new set of inference-time privacy risks: LLMs are fed different types of information from multiple sources in their inputs and are expected to reason about what to share in their outputs, for what purpose and with whom, within a given context. In this work, we draw attention to the highly critical yet overlooked notion of contextual privacy by proposing ConfAIde, a benchmark designed to identify critical weaknesses in the privacy reasoning capabilities of instruction-tuned LLMs. Our experiments show that even the most capable models such as GPT-4 and ChatGPT reveal private information in contexts that humans would not, 39% and 57% of the time, respectively. This leakage persists even when we employ privacy-inducing prompts or chain-of-thought reasoning. Our work underscores the immediate need to explore novel inference-time privacy-preserving approaches, based on reasoning and theory of mind.

Related papers

• Mind the Privacy Unit! User-Level Differential Privacy for Language Model Fine-Tuning [62.224804688233]
  differential privacy (DP) offers a promising solution by ensuring models are 'almost indistinguishable' with or without any particular privacy unit. We study user-level DP motivated by applications where it necessary to ensure uniform privacy protection across users.
  arXiv  Detail & Related papers  (2024-06-20T13:54:32Z)
• No Free Lunch Theorem for Privacy-Preserving LLM Inference [30.554456047738295]
  This study develops a framework for inferring privacy-protected Large Language Models (LLMs) It lays down a solid theoretical basis for examining the interplay between privacy preservation and utility.
  arXiv  Detail & Related papers  (2024-05-31T08:22:53Z)
• Unmemorization in Large Language Models via Self-Distillation and Deliberate Imagination [58.36408867180233]
  Large Language Models (LLMs) struggle with crucial issues of privacy violation and unwanted exposure of sensitive data. We introduce a novel approach termed deliberate imagination in the context of LLM unlearning. Our results demonstrate the usefulness of this approach across different models and sizes, and also with parameter-efficient fine-tuning.
  arXiv  Detail & Related papers  (2024-02-15T16:21:14Z)
• RECALL: A Benchmark for LLMs Robustness against External Counterfactual Knowledge [69.79676144482792]
  This study aims to evaluate the ability of LLMs to distinguish reliable information from external knowledge. Our benchmark consists of two tasks, Question Answering and Text Generation, and for each task, we provide models with a context containing counterfactual information.
  arXiv  Detail & Related papers  (2023-11-14T13:24:19Z)
• PrivLM-Bench: A Multi-level Privacy Evaluation Benchmark for Language Models [42.20437015301152]
  We present PrivLM-Bench, a benchmark for evaluating the privacy leakage of language models (LMs) Instead of only reporting DP parameters, PrivLM-Bench sheds light on the neglected inference data privacy during actual usage. We conduct extensive experiments on three datasets of GLUE for mainstream LMs.
  arXiv  Detail & Related papers  (2023-11-07T14:55:52Z)
• Beyond Memorization: Violating Privacy Via Inference with Large Language Models [2.9373912230684565]
  We present the first comprehensive study on the capabilities of pretrained language models to infer personal attributes from text. Our findings highlight that current LLMs can infer personal data at a previously unattainable scale.
  arXiv  Detail & Related papers  (2023-10-11T08:32:46Z)
• Large Language Models Can Be Good Privacy Protection Learners [53.07930843882592]
  We introduce Privacy Protection Language Models (PPLM), a novel paradigm for fine-tuning language models. Our work offers a theoretical analysis for model design and delves into various techniques such as corpus curation, penalty-based unlikelihood in training loss, and instruction-based tuning. In particular, instruction tuning with both positive and negative examples, stands out as a promising method, effectively protecting private data while enhancing the model's knowledge.
  arXiv  Detail & Related papers  (2023-10-03T22:37:01Z)
• Privacy Implications of Retrieval-Based Language Models [26.87950501433784]
  We present the first study of privacy risks in retrieval-based LMs, particularly $k$NN-LMs. We find that $k$NN-LMs are more susceptible to leaking private information from their private datastore than parametric models.
  arXiv  Detail & Related papers  (2023-05-24T08:37:27Z)
• Multi-step Jailbreaking Privacy Attacks on ChatGPT [47.10284364632862]
  We study the privacy threats from OpenAI's ChatGPT and the New Bing enhanced by ChatGPT. We conduct extensive experiments to support our claims and discuss LLMs' privacy implications.
  arXiv  Detail & Related papers  (2023-04-11T13:05:04Z)
• On Privacy and Confidentiality of Communications in Organizational Graphs [3.5270468102327004]
  This work shows how confidentiality is distinct from privacy in an enterprise context. It aims to formulate an approach to preserving confidentiality while leveraging principles from differential privacy.
  arXiv  Detail & Related papers  (2021-05-27T19:45:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.