Does Differential Privacy Impact Bias in Pretrained NLP Models?

• URL: http://arxiv.org/abs/2410.18749v1
• Date: Thu, 24 Oct 2024 13:59:03 GMT
• Title: Does Differential Privacy Impact Bias in Pretrained NLP Models?
• Authors: Md. Khairul Islam, Andrew Wang, Tianhao Wang, Yangfeng Ji, Judy Fox, Jieyu Zhao,
• Abstract summary: Differential privacy (DP) is applied when fine-tuning pre-trained large language models (LLMs) to limit leakage of training examples. We show the impact of DP on bias in LLMs through empirical analysis. Our results also show that the impact of DP on bias is not only affected by the privacy protection level but also the underlying distribution of the dataset.
• Score: 24.63118058112066
• License:
• Abstract: Differential privacy (DP) is applied when fine-tuning pre-trained large language models (LLMs) to limit leakage of training examples. While most DP research has focused on improving a model's privacy-utility tradeoff, some find that DP can be unfair to or biased against underrepresented groups. In this work, we show the impact of DP on bias in LLMs through empirical analysis. Differentially private training can increase the model bias against protected groups w.r.t AUC-based bias metrics. DP makes it more difficult for the model to differentiate between the positive and negative examples from the protected groups and other groups in the rest of the population. Our results also show that the impact of DP on bias is not only affected by the privacy protection level but also the underlying distribution of the dataset.

Related papers

• A Systematic and Formal Study of the Impact of Local Differential Privacy on Fairness: Preliminary Results [5.618541935188389]
  Differential privacy (DP) is the predominant solution for privacy-preserving Machine learning (ML) algorithms. Recent experimental studies have shown that local DP can impact ML prediction for different subgroups of individuals. We study how the fairness of the decisions made by the ML model changes under local DP for different levels of privacy and data distributions.
  arXiv  Detail & Related papers  (2024-05-23T15:54:03Z)
• Incentives in Private Collaborative Machine Learning [56.84263918489519]
  Collaborative machine learning involves training models on data from multiple parties. We introduce differential privacy (DP) as an incentive. We empirically demonstrate the effectiveness and practicality of our approach on synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2024-04-02T06:28:22Z)
• Pre-training Differentially Private Models with Limited Public Data [54.943023722114134]
  differential privacy (DP) is a prominent method to gauge the degree of security provided to the models. DP is yet not capable of protecting a substantial portion of the data used during the initial pre-training stage. We develop a novel DP continual pre-training strategy using only 10% of public data. Our strategy can achieve DP accuracy of 41.5% on ImageNet-21k, as well as non-DP accuracy of 55.7% and and 60.0% on downstream tasks Places365 and iNaturalist-2021.
  arXiv  Detail & Related papers  (2024-02-28T23:26:27Z)
• LLM-based Privacy Data Augmentation Guided by Knowledge Distillation with a Distribution Tutor for Medical Text Classification [67.92145284679623]
  We propose a DP-based tutor that models the noised private distribution and controls samples' generation with a low privacy cost. We theoretically analyze our model's privacy protection and empirically verify our model.
  arXiv  Detail & Related papers  (2024-02-26T11:52:55Z)
• De-amplifying Bias from Differential Privacy in Language Model Fine-tuning [10.847913815093179]
  Fairness and privacy are two important values machine learning (ML) practitioners often seek to operationalize in models. We show that DP amplifies gender, racial, and religious bias when fine-tuning large language models. We demonstrate that Counterfactual Data Augmentation, a known method for addressing bias, also mitigates bias amplification by DP.
  arXiv  Detail & Related papers  (2024-02-07T00:30:58Z)
• Large Scale Transfer Learning for Differentially Private Image Classification [51.10365553035979]
  Differential Privacy (DP) provides a formal framework for training machine learning models with individual example level privacy. Private training using DP-SGD protects against leakage by injecting noise into individual example gradients. While this result is quite appealing, the computational cost of training large-scale models with DP-SGD is substantially higher than non-private training.
  arXiv  Detail & Related papers  (2022-05-06T01:22:20Z)
• Cross Pairwise Ranking for Unbiased Item Recommendation [57.71258289870123]
  We develop a new learning paradigm named Cross Pairwise Ranking (CPR) CPR achieves unbiased recommendation without knowing the exposure mechanism. We prove in theory that this way offsets the influence of user/item propensity on the learning.
  arXiv  Detail & Related papers  (2022-04-26T09:20:27Z)
• DP-SGD vs PATE: Which Has Less Disparate Impact on GANs? [0.0]
  We compare GANs trained with the two best-known DP frameworks for deep learning, DP-SGD, and PATE, in different data imbalance settings. Our experiments consistently show that for PATE, unlike DP-SGD, the privacy-utility trade-off is not monotonically decreasing.
  arXiv  Detail & Related papers  (2021-11-26T17:25:46Z)
• Robin Hood and Matthew Effects -- Differential Privacy Has Disparate Impact on Synthetic Data [3.2345600015792564]
  We analyze the impact of Differential Privacy on generative models. We show that DP results in opposite size distributions in the generated synthetic data. We call for caution when analyzing or training a model on synthetic data.
  arXiv  Detail & Related papers  (2021-09-23T15:14:52Z)
• DP-SGD vs PATE: Which Has Less Disparate Impact on Model Accuracy? [1.3238373064156095]
  We show that application of differential privacy, specifically the DP-SGD algorithm, has a disparate impact on different sub-groups in the population. We compare PATE, another mechanism for training deep learning models using differential privacy, with DP-SGD in terms of fairness.
  arXiv  Detail & Related papers  (2021-06-22T20:37:12Z)
• DP-InstaHide: Provably Defusing Poisoning and Backdoor Attacks with Differentially Private Data Augmentations [54.960853673256]
  We show that strong data augmentations, such as mixup and random additive noise, nullify poison attacks while enduring only a small accuracy trade-off. A rigorous analysis of DP-InstaHide shows that mixup does indeed have privacy advantages, and that training with k-way mixup provably yields at least k times stronger DP guarantees than a naive DP mechanism.
  arXiv  Detail & Related papers  (2021-03-02T23:07:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.