The Privacy-Hallucination Tradeoff in Differentially Private Language Models

• URL: http://arxiv.org/abs/2609.00492v1
• Date: Mon, 31 Aug 2026 23:39:03 GMT
• Title: The Privacy-Hallucination Tradeoff in Differentially Private Language Models
• Abstract summary: We investigate a privacy-hallucination tradeoff in differentially private (DP) language models.<n>We show that models pre-trained or fine-tuned with DP tend to produce more hallucinations than non-DP counterparts.
• Score: 19.291584742880953
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Both privacy and factual accuracy are paramount in high-stakes domains like healthcare. Concerningly, we uncover and investigate a privacy-hallucination tradeoff in differentially private (DP) language models. First, we empirically show that models pre-trained or fine-tuned with DP tend to produce more hallucinations than non-DP counterparts, with increased severity as the privacy budget grows stricter. Second, we investigate model properties driving this tradeoff, demonstrating that DP mechanisms flatten output distributions, potentially redistributing probability mass toward factually incorrect alternatives. Third, through experiments where we control fact frequency in training data, we characterize how information frequency can reduce hallucination risks in DP models. Overall, our findings underscore the need for more nuanced privacy-preserving interventions that offer rigorous privacy guarantees without compromising factual accuracy.

Related papers

• On the MIA Vulnerability Gap Between Private GANs and Diffusion Models [51.53790101362898]
  Generative Adversarial Networks (GANs) and diffusion models have emerged as leading approaches for high-quality image synthesis.<n>We present the first unified theoretical and empirical analysis of the privacy risks faced by differentially private generative models.
  arXiv  Detail & Related papers  (2025-09-03T14:18:22Z)
• Machine Learning with Privacy for Protected Attributes [56.44253915927481]
  We refine the definition of differential privacy (DP) to create a more general and flexible framework that we call feature differential privacy (FDP)<n>Our definition is simulation-based and allows for both addition/removal and replacement variants of privacy, and can handle arbitrary separation of protected and non-protected features.<n>We apply our framework to various machine learning tasks and show that it can significantly improve the utility of DP-trained models when public features are available.
  arXiv  Detail & Related papers  (2025-06-24T17:53:28Z)
• Differential privacy enables fair and accurate AI-based analysis of speech disorders while protecting patient data [10.6135892856374]
  This study is the first to investigate differential privacy in pathological speech data.<n>It focuses on the trade-offs between privacy, diagnostic accuracy, and fairness.<n>Our results establish that DP can balance privacy and utility in speech disorder detection.
  arXiv  Detail & Related papers  (2024-09-27T18:25:54Z)
• De-amplifying Bias from Differential Privacy in Language Model Fine-tuning [10.847913815093179]
  Fairness and privacy are two important values machine learning (ML) practitioners often seek to operationalize in models. We show that DP amplifies gender, racial, and religious bias when fine-tuning large language models. We demonstrate that Counterfactual Data Augmentation, a known method for addressing bias, also mitigates bias amplification by DP.
  arXiv  Detail & Related papers  (2024-02-07T00:30:58Z)
• Privacy Constrained Fairness Estimation for Decision Trees [2.9906966931843093]
  Measuring the fairness of any AI model requires the sensitive attributes of the individuals in the dataset. We propose a novel method, dubbed Privacy-Aware Fairness Estimation of Rules (PAFER) We show that using the Laplacian mechanism, the method is able to estimate SP with low error while guaranteeing the privacy of the individuals in the dataset with high certainty.
  arXiv  Detail & Related papers  (2023-12-13T14:54:48Z)
• Unlocking Accuracy and Fairness in Differentially Private Image Classification [43.53494043189235]
  Differential privacy (DP) is considered the gold standard framework for privacy-preserving training. We show that pre-trained foundation models fine-tuned with DP can achieve similar accuracy to non-private classifiers.
  arXiv  Detail & Related papers  (2023-08-21T17:42:33Z)
• Private, fair and accurate: Training large-scale, privacy-preserving AI models in medical imaging [47.99192239793597]
  We evaluated the effect of privacy-preserving training of AI models regarding accuracy and fairness compared to non-private training. Our study shows that -- under the challenging realistic circumstances of a real-life clinical dataset -- the privacy-preserving training of diagnostic deep learning models is possible with excellent diagnostic accuracy and fairness.
  arXiv  Detail & Related papers  (2023-02-03T09:49:13Z)
• DP2-Pub: Differentially Private High-Dimensional Data Publication with Invariant Post Randomization [58.155151571362914]
  We propose a differentially private high-dimensional data publication mechanism (DP2-Pub) that runs in two phases. splitting attributes into several low-dimensional clusters with high intra-cluster cohesion and low inter-cluster coupling helps obtain a reasonable privacy budget. We also extend our DP2-Pub mechanism to the scenario with a semi-honest server which satisfies local differential privacy.
  arXiv  Detail & Related papers  (2022-08-24T17:52:43Z)
• Just Fine-tune Twice: Selective Differential Privacy for Large Language Models [69.66654761324702]
  We propose a simple yet effective just-fine-tune-twice privacy mechanism to achieve SDP for large Transformer-based language models. Experiments show that our models achieve strong performance while staying robust to the canary insertion attack.
  arXiv  Detail & Related papers  (2022-04-15T22:36:55Z)
• Don't Generate Me: Training Differentially Private Generative Models with Sinkhorn Divergence [73.14373832423156]
  We propose DP-Sinkhorn, a novel optimal transport-based generative method for learning data distributions from private data with differential privacy. Unlike existing approaches for training differentially private generative models, we do not rely on adversarial objectives.
  arXiv  Detail & Related papers  (2021-11-01T18:10:21Z)
• Smoothed Differential Privacy [55.415581832037084]
  Differential privacy (DP) is a widely-accepted and widely-applied notion of privacy based on worst-case analysis. In this paper, we propose a natural extension of DP following the worst average-case idea behind the celebrated smoothed analysis. We prove that any discrete mechanism with sampling procedures is more private than what DP predicts, while many continuous mechanisms with sampling procedures are still non-private under smoothed DP.
  arXiv  Detail & Related papers  (2021-07-04T06:55:45Z)
• Accuracy, Interpretability, and Differential Privacy via Explainable Boosting [22.30100748652558]
  We show that adding differential privacy to Explainable Boosting Machines (EBMs) yields state-of-the-art accuracy while protecting privacy. Our experiments on multiple classification and regression datasets show that DP-EBM models suffer surprisingly little accuracy loss even with strong differential privacy guarantees.
  arXiv  Detail & Related papers  (2021-06-17T17:33:00Z)
• Chasing Your Long Tails: Differentially Private Prediction in Health Care Settings [34.26542589537452]
  Methods for differentially private (DP) learning provide a general-purpose approach to learn models with privacy guarantees. Modern methods for DP learning ensure privacy through mechanisms that censor information judged as too unique. We use state-of-the-art methods for DP learning to train privacy-preserving models in clinical prediction tasks.
  arXiv  Detail & Related papers  (2020-10-13T19:56:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.