KART: Privacy Leakage Framework of Language Models Pre-trained with Clinical Records

• URL: http://arxiv.org/abs/2101.00036v1
• Date: Thu, 31 Dec 2020 19:06:18 GMT
• Title: KART: Privacy Leakage Framework of Language Models Pre-trained with Clinical Records
• Authors: Yuta Nakamura (1 and 2), Shouhei Hanaoka (3), Yukihiro Nomura (4), Naoto Hayashi (4), Osamu Abe (1 and 3), Shuntaro Yada (2), Shoko Wakamiya (2), Eiji Aramaki (2) ((1) The University of Tokyo, (2) Nara Institute of Science and Technology, (3) The Department of Radiology, The University of Tokyo Hospital, (4) The Department of Computational Diagnostic Radiology and Preventive Medicine, The University of Tokyo Hospital)
• Abstract summary: We empirically evaluated the privacy risk of language models, using several BERT models pre-trained with MIMIC-III corpus. BERT models were probably low-risk because the Top-100 accuracy of each attack was far below expected by chance. We formalized various privacy leakage scenarios under a universal novel framework named Knowledge, Anonymization, Resource, and Target (KART) framework.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Nowadays, mainstream natural language pro-cessing (NLP) is empowered by pre-trained language models. In the biomedical domain, only models pre-trained with anonymized data have been published. This policy is acceptable, but there are two questions: Can the privacy policy of language models be different from that of data? What happens if private language models are accidentally made public? We empirically evaluated the privacy risk of language models, using several BERT models pre-trained with MIMIC-III corpus in different data anonymity and corpus sizes. We simulated model inversion attacks to obtain the clinical information of target individuals, whose full names are already known to attackers. The BERT models were probably low-risk because the Top-100 accuracy of each attack was far below expected by chance. Moreover, most privacy leakage situations have several common primary factors; therefore, we formalized various privacy leakage scenarios under a universal novel framework named Knowledge, Anonymization, Resource, and Target (KART) framework. The KART framework helps parameterize complex privacy leakage scenarios and simplifies the comprehensive evaluation. Since the concept of the KART framework is domain agnostic, it can contribute to the establishment of privacy guidelines of language models beyond the biomedical domain.

Related papers

• Current State in Privacy-Preserving Text Preprocessing for Domain-Agnostic NLP [0.0]
  Modern large language models require a huge amount of data to learn linguistic variations.<n>It is possible to extract private information from such language models.<n>This report focuses on a few approaches for domain-agnostic NLP tasks.
  arXiv  Detail & Related papers  (2025-08-05T08:26:45Z)
• Understanding and Mitigating Cross-lingual Privacy Leakage via Language-specific and Universal Privacy Neurons [17.557961521354766]
  This work investigates the information flow of cross-lingual privacy leakage.<n>We identify privacy-universal neurons and language-specific privacy neurons.<n>By deactivating these neurons, the cross-lingual privacy leakage risk is reduced by 23.3%-31.6%.
  arXiv  Detail & Related papers  (2025-06-01T00:10:30Z)
• PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Action [54.11479432110771]
  PrivacyLens is a novel framework designed to extend privacy-sensitive seeds into expressive vignettes and further into agent trajectories. We instantiate PrivacyLens with a collection of privacy norms grounded in privacy literature and crowdsourced seeds. State-of-the-art LMs, like GPT-4 and Llama-3-70B, leak sensitive information in 25.68% and 38.69% of cases, even when prompted with privacy-enhancing instructions.
  arXiv  Detail & Related papers  (2024-08-29T17:58:38Z)
• Privacy Backdoors: Enhancing Membership Inference through Poisoning Pre-trained Models [112.48136829374741]
  In this paper, we unveil a new vulnerability: the privacy backdoor attack. When a victim fine-tunes a backdoored model, their training data will be leaked at a significantly higher rate than if they had fine-tuned a typical model. Our findings highlight a critical privacy concern within the machine learning community and call for a reevaluation of safety protocols in the use of open-source pre-trained models.
  arXiv  Detail & Related papers  (2024-04-01T16:50:54Z)
• Membership Inference Attacks and Privacy in Topic Modeling [3.503833571450681]
  We propose an attack against topic models that can confidently identify members of the training data. We propose a framework for private topic modeling that incorporates DP vocabulary selection as a pre-processing step.
  arXiv  Detail & Related papers  (2024-03-07T12:43:42Z)
• Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory [82.7042006247124]
  We show that even the most capable AI models reveal private information in contexts that humans would not, 39% and 57% of the time, respectively. Our work underscores the immediate need to explore novel inference-time privacy-preserving approaches, based on reasoning and theory of mind.
  arXiv  Detail & Related papers  (2023-10-27T04:15:30Z)
• FLTrojan: Privacy Leakage Attacks against Federated Language Models Through Selective Weight Tampering [2.2194815687410627]
  We show how a malicious client can leak the privacy-sensitive data of some other users in FL even without any cooperation from the server. Our best-performing method improves the membership inference recall by 29% and achieves up to 71% private data reconstruction.
  arXiv  Detail & Related papers  (2023-10-24T19:50:01Z)
• Locally Differentially Private Document Generation Using Zero Shot Prompting [61.20953109732442]
  We propose a locally differentially private mechanism called DP-Prompt to counter author de-anonymization attacks. When DP-Prompt is used with a powerful language model like ChatGPT (gpt-3.5), we observe a notable reduction in the success rate of de-anonymization attacks.
  arXiv  Detail & Related papers  (2023-10-24T18:25:13Z)
• PrivacyMind: Large Language Models Can Be Contextual Privacy Protection Learners [81.571305826793]
  We introduce Contextual Privacy Protection Language Models (PrivacyMind) Our work offers a theoretical analysis for model design and benchmarks various techniques. In particular, instruction tuning with both positive and negative examples stands out as a promising method.
  arXiv  Detail & Related papers  (2023-10-03T22:37:01Z)
• Can Language Models be Instructed to Protect Personal Information? [30.187731765653428]
  We introduce PrivQA -- a benchmark to assess the privacy/utility trade-off when a model is instructed to protect specific categories of personal information in a simulated scenario. We find that adversaries can easily circumvent these protections with simple jailbreaking methods through textual and/or image inputs. We believe PrivQA has the potential to support the development of new models with improved privacy protections, as well as the adversarial robustness of these protections.
  arXiv  Detail & Related papers  (2023-10-03T17:30:33Z)
• Does CLIP Know My Face? [31.21910897081894]
  We introduce a novel method to assess privacy for multi-modal models, specifically vision-language models like CLIP. The proposed Identity Inference Attack (IDIA) reveals whether an individual was included in the training data by querying the model with images of the same person. Our results highlight the need for stronger privacy protection in large-scale models and suggest that IDIAs can be used to prove the unauthorized use of data for training and to enforce privacy laws.
  arXiv  Detail & Related papers  (2022-09-15T14:48:50Z)
• Just Fine-tune Twice: Selective Differential Privacy for Large Language Models [69.66654761324702]
  We propose a simple yet effective just-fine-tune-twice privacy mechanism to achieve SDP for large Transformer-based language models. Experiments show that our models achieve strong performance while staying robust to the canary insertion attack.
  arXiv  Detail & Related papers  (2022-04-15T22:36:55Z)
• Selective Differential Privacy for Language Modeling [36.64464956102432]
  Previous work has attempted to tackle this challenge by training RNN-based language models with differential privacy guarantees. We propose a new privacy notion, selective differential privacy, to provide rigorous privacy guarantees on the sensitive portion of the data. Experiments on both language modeling and dialog system building show that the proposed privacy-preserving mechanism achieves better utilities.
  arXiv  Detail & Related papers  (2021-08-30T01:11:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.