Benchmarking Empirical Privacy Protection for Adaptations of Large Language Models

• URL: http://arxiv.org/abs/2606.09401v1
• Date: Mon, 08 Jun 2026 12:21:02 GMT
• Title: Benchmarking Empirical Privacy Protection for Adaptations of Large Language Models
• Abstract summary: We investigate privacy risks under DP adaptations in large language models (LLMs)<n>We benchmark these risks using state-of-the-art attacks such as robust membership inference and canary data extraction.<n>We find that parameter-efficient fine-tuning methods, such as LoRA, achieve the highest empirical privacy protection for OOD data.
• Score: 35.164227002323706
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent work has applied differential privacy (DP) to adapt large language models (LLMs) for sensitive applications, offering theoretical guarantees. However, its practical effectiveness remains unclear, partly due to LLM pretraining, where overlaps and interdependencies with adaptation data can undermine privacy despite DP efforts. To analyze this issue in practice, we investigate privacy risks under DP adaptations in LLMs using state-of-the-art attacks such as robust membership inference and canary data extraction. We benchmark these risks by systematically varying the adaptation data distribution, from exact overlaps with pretraining data, through in-distribution (IID) cases, to entirely out-of-distribution (OOD) examples. Additionally, we evaluate how different adaptation methods and different privacy regimes impact the vulnerability. Our results show that distribution shifts strongly influence privacy vulnerability: the closer the adaptation data is to the pretraining distribution, the higher the practical privacy risk at the same theoretical guarantee, even without direct data overlap. We find that parameter-efficient fine-tuning methods, such as LoRA, achieve the highest empirical privacy protection for OOD data. Our benchmark identifies key factors for achieving practical privacy in DP LLM adaptation, providing actionable insights for deploying customized models in sensitive settings. Looking forward, we propose a structured framework for holistic privacy assessment beyond adaptation privacy, to identify and evaluate risks across the full pretrain-adapt pipeline of LLMs.

Related papers

• On the MIA Vulnerability Gap Between Private GANs and Diffusion Models [51.53790101362898]
  Generative Adversarial Networks (GANs) and diffusion models have emerged as leading approaches for high-quality image synthesis.<n>We present the first unified theoretical and empirical analysis of the privacy risks faced by differentially private generative models.
  arXiv  Detail & Related papers  (2025-09-03T14:18:22Z)
• Can Differentially Private Fine-tuning LLMs Protect Against Privacy Attacks? [8.189149471520542]
  Fine-tuning large language models (LLMs) has become an essential strategy for adapting them to specialized tasks.<n>Although differential privacy (DP) offers strong theoretical guarantees against such leakage, its empirical privacy effectiveness on LLMs remains unclear.<n>This paper systematically investigates the impact of DP across fine-tuning methods and privacy budgets.
  arXiv  Detail & Related papers  (2025-04-28T05:34:53Z)
• Evaluating Differential Privacy on Correlated Datasets Using Pointwise Maximal Leakage [38.4830633082184]
  Data-driven advancements pose substantial risks to privacy.<n> differential privacy has become a cornerstone in privacy preservation efforts.<n>Our work aims to foster a deeper understanding of subtle privacy risks and highlight the need for the development of more effective privacy-preserving mechanisms.
  arXiv  Detail & Related papers  (2025-02-08T10:30:45Z)
• Communication-Efficient and Privacy-Adaptable Mechanism for Federated Learning [54.20871516148981]
  We introduce the Communication-Efficient and Privacy-Adaptable Mechanism (CEPAM)<n>CEPAM achieves communication efficiency and privacy protection simultaneously.<n>We theoretically analyze the privacy guarantee of CEPAM and investigate the trade-offs among user privacy and accuracy of CEPAM.
  arXiv  Detail & Related papers  (2025-01-21T11:16:05Z)
• Pseudo-Probability Unlearning: Towards Efficient and Privacy-Preserving Machine Unlearning [59.29849532966454]
  We propose PseudoProbability Unlearning (PPU), a novel method that enables models to forget data to adhere to privacy-preserving manner. Our method achieves over 20% improvements in forgetting error compared to the state-of-the-art.
  arXiv  Detail & Related papers  (2024-11-04T21:27:06Z)
• Enhancing Feature-Specific Data Protection via Bayesian Coordinate Differential Privacy [55.357715095623554]
  Local Differential Privacy (LDP) offers strong privacy guarantees without requiring users to trust external parties. We propose a Bayesian framework, Bayesian Coordinate Differential Privacy (BCDP), that enables feature-specific privacy quantification.
  arXiv  Detail & Related papers  (2024-10-24T03:39:55Z)
• Noisy Neighbors: Efficient membership inference attacks against LLMs [2.666596421430287]
  This paper introduces an efficient methodology that generates textitnoisy neighbors for a target sample by adding noise in the embedding space. Our findings demonstrate that this approach closely matches the effectiveness of employing shadow models, showing its usability in practical privacy auditing scenarios.
  arXiv  Detail & Related papers  (2024-06-24T12:02:20Z)
• Flexible Differentially Private Vertical Federated Learning with Adaptive Feature Embeddings [24.36847069007795]
  Vertical federated learning (VFL) has stimulated concerns about the imperfection in privacy protection. This paper studies the delicate equilibrium between data privacy and task utility goals of VFL under differential privacy (DP) We propose a flexible and generic approach that decouples the two goals and addresses them successively.
  arXiv  Detail & Related papers  (2023-07-26T04:40:51Z)
• A Differentially Private Weighted Empirical Risk Minimization Procedure and its Application to Outcome Weighted Learning [4.322221694511603]
  Differential privacy (DP) is an appealing framework for addressing data privacy issues. DP provides mathematically provable bounds on the privacy loss incurred when releasing information from sensitive data. We propose the first differentially private algorithm for general wERM, with theoretical DP guarantees.
  arXiv  Detail & Related papers  (2023-07-24T21:03:25Z)
• Theoretically Principled Federated Learning for Balancing Privacy and Utility [61.03993520243198]
  We propose a general learning framework for the protection mechanisms that protects privacy via distorting model parameters. It can achieve personalized utility-privacy trade-off for each model parameter, on each client, at each communication round in federated learning.
  arXiv  Detail & Related papers  (2023-05-24T13:44:02Z)
• FedLAP-DP: Federated Learning by Sharing Differentially Private Loss Approximations [53.268801169075836]
  We propose FedLAP-DP, a novel privacy-preserving approach for federated learning. A formal privacy analysis demonstrates that FedLAP-DP incurs the same privacy costs as typical gradient-sharing schemes. Our approach presents a faster convergence speed compared to typical gradient-sharing methods.
  arXiv  Detail & Related papers  (2023-02-02T12:56:46Z)
• Differentially Private Federated Learning with Laplacian Smoothing [72.85272874099644]
  Federated learning aims to protect data privacy by collaboratively learning a model without sharing private data among users. An adversary may still be able to infer the private training data by attacking the released model. Differential privacy provides a statistical protection against such attacks at the price of significantly degrading the accuracy or utility of the trained models.
  arXiv  Detail & Related papers  (2020-05-01T04:28:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.