Underestimated Privacy Risks for Minority Populations in Large Language Model Unlearning

• URL: http://arxiv.org/abs/2412.08559v2
• Date: Sun, 19 Jan 2025 03:05:22 GMT
• Title: Underestimated Privacy Risks for Minority Populations in Large Language Model Unlearning
• Authors: Rongzhe Wei, Mufei Li, Mohsen Ghassemi, Eleonora Kreačić, Yifan Li, Xiang Yue, Bo Li, Vamsi K. Potluru, Pan Li, Eli Chien,
• Abstract summary: We argue that unlearning should be considered in the worst-case scenario from the privacy perspective.<n>Minority groups experience at least 20% more privacy leakage in most cases across six unlearning approaches.
• Score: 20.018234150653885
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models are trained on extensive datasets that often contain sensitive, human-generated information, raising significant concerns about privacy breaches. While certified unlearning approaches offer strong privacy guarantees, they rely on restrictive model assumptions that are not applicable to LLMs. As a result, various unlearning heuristics have been proposed, with the associated privacy risks assessed only empirically. The standard evaluation pipelines typically randomly select data for removal from the training set, apply unlearning techniques, and use membership inference attacks to compare the unlearned models against models retrained without the to-be-unlearned data. However, since every data point is subject to the right to be forgotten, unlearning should be considered in the worst-case scenario from the privacy perspective. Prior work shows that data outliers may exhibit higher memorization effects. Intuitively, they are harder to be unlearn and thus the privacy risk of unlearning them is underestimated in the current evaluation. In this paper, we leverage minority data to identify such a critical flaw in previously widely adopted evaluations. We substantiate this claim through carefully designed experiments, including unlearning canaries related to minority groups, inspired by privacy auditing literature. Using personally identifiable information as a representative minority identifier, we demonstrate that minority groups experience at least 20% more privacy leakage in most cases across six unlearning approaches, three MIAs, three benchmark datasets, and two LLMs of different scales. Given that the right to be forgotten should be upheld for every individual, we advocate for a more rigorous evaluation of LLM unlearning methods. Our minority-aware evaluation framework represents an initial step toward ensuring more equitable assessments of LLM unlearning efficacy.

Related papers

• Membership Inference Attacks fueled by Few-Short Learning to detect privacy leakage tackling data integrity [7.8973037023478785]
  Deep learning models memorize parts of their training data, creating a privacy leakage. We propose a Few-Shot learning based MIA, coined as the FeS-MIA model, which eases the evaluation of the privacy breach of a deep learning model. We also propose an interpretable quantitative and qualitative measure of privacy, referred to as Log-MIA measure.
  arXiv  Detail & Related papers  (2025-03-12T13:09:43Z)
• Benchmarking Vision Language Model Unlearning via Fictitious Facial Identity Dataset [94.13848736705575]
  We introduce Facial Identity Unlearning Benchmark (FIUBench), a novel VLM unlearning benchmark designed to robustly evaluate the effectiveness of unlearning algorithms. We apply a two-stage evaluation pipeline that is designed to precisely control the sources of information and their exposure levels. Through the evaluation of four baseline VLM unlearning algorithms within FIUBench, we find that all methods remain limited in their unlearning performance.
  arXiv  Detail & Related papers  (2024-11-05T23:26:10Z)
• Investigating Privacy Bias in Training Data of Language Models [1.3167450470598043]
  A privacy bias refers to the skew in the appropriateness of information flows within a given context. This skew may either align with existing expectations or signal a symptom of systemic issues. We present a novel approach to assess the privacy biases using a contextual integrity-based methodology.
  arXiv  Detail & Related papers  (2024-09-05T17:50:31Z)
• Evaluating Implicit Bias in Large Language Models by Attacking From a Psychometric Perspective [66.34066553400108]
  We conduct a rigorous evaluation of large language models' implicit bias towards certain demographics. Inspired by psychometric principles, we propose three attack approaches, i.e., Disguise, Deception, and Teaching. Our methods can elicit LLMs' inner bias more effectively than competitive baselines.
  arXiv  Detail & Related papers  (2024-06-20T06:42:08Z)
• Towards Effective Evaluations and Comparisons for LLM Unlearning Methods [97.2995389188179]
  This paper seeks to refine the evaluation of machine unlearning for large language models. It addresses two key challenges -- the robustness of evaluation metrics and the trade-offs between competing goals.
  arXiv  Detail & Related papers  (2024-06-13T14:41:00Z)
• Unveiling the Misuse Potential of Base Large Language Models via In-Context Learning [61.2224355547598]
  Open-sourcing of large language models (LLMs) accelerates application development, innovation, and scientific progress. Our investigation exposes a critical oversight in this belief. By deploying carefully designed demonstrations, our research demonstrates that base LLMs could effectively interpret and execute malicious instructions.
  arXiv  Detail & Related papers  (2024-04-16T13:22:54Z)
• Privacy Backdoors: Enhancing Membership Inference through Poisoning Pre-trained Models [112.48136829374741]
  In this paper, we unveil a new vulnerability: the privacy backdoor attack. When a victim fine-tunes a backdoored model, their training data will be leaked at a significantly higher rate than if they had fine-tuned a typical model. Our findings highlight a critical privacy concern within the machine learning community and call for a reevaluation of safety protocols in the use of open-source pre-trained models.
  arXiv  Detail & Related papers  (2024-04-01T16:50:54Z)
• Responsible Active Learning via Human-in-the-loop Peer Study [88.01358655203441]
  We propose a responsible active learning method, namely Peer Study Learning (PSL), to simultaneously preserve data privacy and improve model stability. We first introduce a human-in-the-loop teacher-student architecture to isolate unlabelled data from the task learner (teacher) on the cloud-side. During training, the task learner instructs the light-weight active learner which then provides feedback on the active sampling criterion.
  arXiv  Detail & Related papers  (2022-11-24T13:18:27Z)
• Knowledge Unlearning for Mitigating Privacy Risks in Language Models [31.322818016245087]
  We propose knowledge unlearning as an alternative method to reduce privacy risks for language models. We show that simply applying the unlikelihood training objective to target token sequences is effective at forgetting them. We show that unlearning can give a stronger empirical privacy guarantee in scenarios where the data vulnerable to extraction attacks are known a priori.
  arXiv  Detail & Related papers  (2022-10-04T10:18:11Z)
• Evaluating Machine Unlearning via Epistemic Uncertainty [78.27542864367821]
  This work presents an evaluation of Machine Unlearning algorithms based on uncertainty. This is the first definition of a general evaluation of our best knowledge.
  arXiv  Detail & Related papers  (2022-08-23T09:37:31Z)
• On the Privacy Effect of Data Enhancement via the Lens of Memorization [20.63044895680223]
  We propose to investigate privacy from a new perspective called memorization. Through the lens of memorization, we find that previously deployed MIAs produce misleading results as they are less likely to identify samples with higher privacy risks. We demonstrate that the generalization gap and privacy leakage are less correlated than those of the previous results.
  arXiv  Detail & Related papers  (2022-08-17T13:02:17Z)
• Quantifying and Mitigating Privacy Risks of Contrastive Learning [4.909548818641602]
  We perform the first privacy analysis of contrastive learning through the lens of membership inference and attribute inference. Our results show that contrastive models are less vulnerable to membership inference attacks but more vulnerable to attribute inference attacks compared to supervised models. To remedy this situation, we propose the first privacy-preserving contrastive learning mechanism, namely Talos.
  arXiv  Detail & Related papers  (2021-02-08T11:38:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.