Evaluations of Machine Learning Privacy Defenses are Misleading

• URL: http://arxiv.org/abs/2404.17399v2
• Date: Thu, 5 Sep 2024 16:34:02 GMT
• Title: Evaluations of Machine Learning Privacy Defenses are Misleading
• Authors: Michael Aerni, Jie Zhang, Florian Tramèr,
• Abstract summary: Empirical defenses for machine learning privacy forgo the provable guarantees of differential privacy. We show that prior evaluations fail to characterize the privacy leakage of the most vulnerable samples.
• Score: 25.007083740549845
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Empirical defenses for machine learning privacy forgo the provable guarantees of differential privacy in the hope of achieving higher utility while resisting realistic adversaries. We identify severe pitfalls in existing empirical privacy evaluations (based on membership inference attacks) that result in misleading conclusions. In particular, we show that prior evaluations fail to characterize the privacy leakage of the most vulnerable samples, use weak attacks, and avoid comparisons with practical differential privacy baselines. In 5 case studies of empirical privacy defenses, we find that prior evaluations underestimate privacy leakage by an order of magnitude. Under our stronger evaluation, none of the empirical defenses we study are competitive with a properly tuned, high-utility DP-SGD baseline (with vacuous provable guarantees).

Related papers

• Contextualized Privacy Defense for LLM Agents [84.30907378390512]
  LLM agents increasingly act on users' personal information, yet existing privacy defenses remain limited in both design and adaptability.<n>We propose Contextualized Defense Instructing (CDI), a new privacy defense paradigm.<n>We show that our CDI consistently achieves a better balance between privacy preservation (94.2%) and helpfulness (80.6%) than baselines.
  arXiv  Detail & Related papers  (2026-03-03T13:35:33Z)
• Your Privacy Depends on Others: Collusion Vulnerabilities in Individual Differential Privacy [50.66105844449181]
  Individual Differential Privacy (iDP) promises users control over their privacy, but this promise can be broken in practice.<n>We reveal a previously overlooked vulnerability in sampling-based iDP mechanisms.<n>We propose $(varepsilon_i,_i,overline)$-iDP a privacy contract that uses $$-divergences to provide users with a hard upper bound on their excess vulnerability.
  arXiv  Detail & Related papers  (2026-01-19T10:26:12Z)
• PrivATE: Differentially Private Average Treatment Effect Estimation for Observational Data [49.35645194884526]
  We introduce PrivATE, a practical ATE estimation framework that ensures differential privacy.<n>We design two levels (i.e., label-level and sample-level) of privacy protection in PrivATE to accommodate different privacy requirements.<n>PrivATE effectively balances noise-induced error and matching error, leading to a more accurate estimate of ATE.
  arXiv  Detail & Related papers  (2025-12-16T16:30:07Z)
• PRIVEE: Privacy-Preserving Vertical Federated Learning Against Feature Inference Attacks [20.61987420750636]
  We propose a novel defense mechanism named after the French word privée, meaning "private"<n>PRIvacy-preserving Vertical fEderated lEarning obfuscates confidence scores while preserving critical properties such as relative ranking and inter-score distances.<n>Experiments show that PRIVEE achieves a threefold improvement in privacy protection compared to state-of-the-art defenses.
  arXiv  Detail & Related papers  (2025-12-14T21:05:19Z)
• On the MIA Vulnerability Gap Between Private GANs and Diffusion Models [51.53790101362898]
  Generative Adversarial Networks (GANs) and diffusion models have emerged as leading approaches for high-quality image synthesis.<n>We present the first unified theoretical and empirical analysis of the privacy risks faced by differentially private generative models.
  arXiv  Detail & Related papers  (2025-09-03T14:18:22Z)
• Enhancing Feature-Specific Data Protection via Bayesian Coordinate Differential Privacy [55.357715095623554]
  Local Differential Privacy (LDP) offers strong privacy guarantees without requiring users to trust external parties. We propose a Bayesian framework, Bayesian Coordinate Differential Privacy (BCDP), that enables feature-specific privacy quantification.
  arXiv  Detail & Related papers  (2024-10-24T03:39:55Z)
• Convergent Differential Privacy Analysis for General Federated Learning: the $f$-DP Perspective [57.35402286842029]
  Federated learning (FL) is an efficient collaborative training paradigm with a focus on local privacy. differential privacy (DP) is a classical approach to capture and ensure the reliability of private protections.
  arXiv  Detail & Related papers  (2024-08-28T08:22:21Z)
• A Cautionary Tale: On the Role of Reference Data in Empirical Privacy Defenses [12.34501903200183]
  We propose a baseline defense that enables the utility-privacy tradeoff with respect to both training and reference data to be easily understood. Our experiments show that, surprisingly, it outperforms the most well-studied and current state-of-the-art empirical privacy defenses.
  arXiv  Detail & Related papers  (2023-10-18T17:07:07Z)
• A Randomized Approach for Tight Privacy Accounting [63.67296945525791]
  We propose a new differential privacy paradigm called estimate-verify-release (EVR) EVR paradigm first estimates the privacy parameter of a mechanism, then verifies whether it meets this guarantee, and finally releases the query output. Our empirical evaluation shows the newly proposed EVR paradigm improves the utility-privacy tradeoff for privacy-preserving machine learning.
  arXiv  Detail & Related papers  (2023-04-17T00:38:01Z)
• Privacy in Practice: Private COVID-19 Detection in X-Ray Images (Extended Version) [3.750713193320627]
  We create machine learning models that satisfy Differential Privacy (DP) We evaluate the utility-privacy trade-off more extensively and over stricter privacy budgets. Our results indicate that needed privacy levels might differ based on the task-dependent practical threat from MIAs.
  arXiv  Detail & Related papers  (2022-11-21T13:22:29Z)
• Analyzing Privacy Leakage in Machine Learning via Multiple Hypothesis Testing: A Lesson From Fano [83.5933307263932]
  We study data reconstruction attacks for discrete data and analyze it under the framework of hypothesis testing. We show that if the underlying private data takes values from a set of size $M$, then the target privacy parameter $epsilon$ can be $O(log M)$ before the adversary gains significant inferential power.
  arXiv  Detail & Related papers  (2022-10-24T23:50:12Z)
• On the Statistical Complexity of Estimation and Testing under Privacy Constraints [17.04261371990489]
  We show how to characterize the power of a statistical test under differential privacy in a plug-and-play fashion. We show that maintaining privacy results in a noticeable reduction in performance only when the level of privacy protection is very high. Finally, we demonstrate that the DP-SGLD algorithm, a private convex solver, can be employed for maximum likelihood estimation with a high degree of confidence.
  arXiv  Detail & Related papers  (2022-10-05T12:55:53Z)
• No Free Lunch in "Privacy for Free: How does Dataset Condensation Help Privacy" [75.98836424725437]
  New methods designed to preserve data privacy require careful scrutiny. Failure to preserve privacy is hard to detect, and yet can lead to catastrophic results when a system implementing a privacy-preserving'' method is attacked.
  arXiv  Detail & Related papers  (2022-09-29T17:50:23Z)
• On the Privacy Effect of Data Enhancement via the Lens of Memorization [20.63044895680223]
  We propose to investigate privacy from a new perspective called memorization. Through the lens of memorization, we find that previously deployed MIAs produce misleading results as they are less likely to identify samples with higher privacy risks. We demonstrate that the generalization gap and privacy leakage are less correlated than those of the previous results.
  arXiv  Detail & Related papers  (2022-08-17T13:02:17Z)
• A Critical Review on the Use (and Misuse) of Differential Privacy in Machine Learning [5.769445676575767]
  We review the use of differential privacy (DP) for privacy protection in machine learning (ML) We show that, driven by the aim of preserving the accuracy of the learned models, DP-based ML implementations are so loose that they do not offer the ex ante privacy guarantees of DP.
  arXiv  Detail & Related papers  (2022-06-09T17:13:10Z)
• Private Reinforcement Learning with PAC and Regret Guarantees [69.4202374491817]
  We design privacy preserving exploration policies for episodic reinforcement learning (RL) We first provide a meaningful privacy formulation using the notion of joint differential privacy (JDP) We then develop a private optimism-based learning algorithm that simultaneously achieves strong PAC and regret bounds, and enjoys a JDP guarantee.
  arXiv  Detail & Related papers  (2020-09-18T20:18:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.