Are We There Yet? Timing and Floating-Point Attacks on Differential Privacy Systems

• URL: http://arxiv.org/abs/2112.05307v4
• Date: Wed, 11 Sep 2024 08:56:42 GMT
• Title: Are We There Yet? Timing and Floating-Point Attacks on Differential Privacy Systems
• Abstract summary: We study two implementation flaws in the noise generation commonly used in differentially private (DP) systems. First we examine the Gaussian mechanism's susceptibility to a floating-point representation attack. Second we study discrete counterparts of the Laplace and Gaussian mechanisms that suffer from another side channel: a novel timing attack.
• Score: 18.396937775602808
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Differential privacy is a de facto privacy framework that has seen adoption in practice via a number of mature software platforms. Implementation of differentially private (DP) mechanisms has to be done carefully to ensure end-to-end security guarantees. In this paper we study two implementation flaws in the noise generation commonly used in DP systems. First we examine the Gaussian mechanism's susceptibility to a floating-point representation attack. The premise of this first vulnerability is similar to the one carried out by Mironov in 2011 against the Laplace mechanism. Our experiments show attack's success against DP algorithms, including deep learning models trained using differentially-private stochastic gradient descent. In the second part of the paper we study discrete counterparts of the Laplace and Gaussian mechanisms that were previously proposed to alleviate the shortcomings of floating-point representation of real numbers. We show that such implementations unfortunately suffer from another side channel: a novel timing attack. An observer that can measure the time to draw (discrete) Laplace or Gaussian noise can predict the noise magnitude, which can then be used to recover sensitive attributes. This attack invalidates differential privacy guarantees of systems implementing such mechanisms. We demonstrate that several commonly used, state-of-the-art implementations of differential privacy are susceptible to these attacks. We report success rates up to 92.56% for floating-point attacks on DP-SGD, and up to 99.65% for end-to-end timing attacks on private sum protected with discrete Laplace. Finally, we evaluate and suggest partial mitigations.

Related papers

• Rethinking the Security of DP-SGD: A Corrected Analysis of Differentially Private Machine Learning [16.83879548734828]
  Differentially Private Gradient Descent (DP-SGD) is widely used to protect training data in machine learning.<n>We re-analyze the privacy guarantees of DP-SGD under the Expected-Averaged SGM and Batch-Averaged SGM formulations.<n>Our theoretical results show that these guarantees can be weaker than the standard SGM-based guarantee, implying that the true privacy leakage may exceed the reported guarantee in some regimes.
  arXiv  Detail & Related papers  (2026-05-15T06:04:00Z)
• Efficient privacy loss accounting for subsampling and random allocation [13.186391867041394]
  We consider the privacy amplification properties of a sampling scheme in which a user's data is used in random steps.<n>We show that the privacy-utility trade-off for random allocation is at least as good as that of Poisson subsampling.
  arXiv  Detail & Related papers  (2026-02-19T11:44:25Z)
• LoRA and Privacy: When Random Projections Help (and When They Don't) [55.65932772290123]
  We introduce the (Wishart) projection mechanism, a randomized map of the form $S mapsto M f(S)$ with $M sim W_d (1/r I_d, r)$ and study its differential privacy properties.<n>For vector-valued queries $f$, we prove non-asymptotic DP guarantees without any additive noise, showing that Wishart randomness alone can suffice.<n>For matrix-valued queries, however, we establish a sharp negative result: in the noise-free setting, the mechanism is not DP, and we demonstrate its vulnerability.
  arXiv  Detail & Related papers  (2026-01-29T13:43:37Z)
• GPM: The Gaussian Pancake Mechanism for Planting Undetectable Backdoors in Differential Privacy [4.281902449361707]
  Several incidents of unintended privacy loss have occurred due to numerical issues and inappropriate configurations of Differential Privacy software.<n>We present the Gaussian pancake mechanism (GPM), a novel mechanism that is computationally indistinguishable from the widely used Gaussian mechanism (GM)<n>Unlike the unintentional privacy loss caused by GM's numerical issues, GPM is an adversarial yet undetectable backdoor attack against data privacy.
  arXiv  Detail & Related papers  (2025-09-28T12:14:06Z)
• Perfectly-Private Analog Secure Aggregation in Federated Learning [51.61616734974475]
  In federated learning, multiple parties train models locally and share their parameters with a central server, which aggregates them to update a global model.<n>In this paper, a novel secure parameter aggregation method is proposed that employs the torus rather than a finite field.
  arXiv  Detail & Related papers  (2025-09-10T15:22:40Z)
• Benchmarking Fraud Detectors on Private Graph Data [70.4654745317714]
  Currently, many types of fraud are managed in part by automated detection algorithms that operate over graphs.<n>We consider the scenario where a data holder wishes to outsource development of fraud detectors to third parties.<n>Third parties submit their fraud detectors to the data holder, who evaluates these algorithms on a private dataset and then publicly communicates the results.<n>We propose a realistic privacy attack on this system that allows an adversary to de-anonymize individuals' data based only on the evaluation results.
  arXiv  Detail & Related papers  (2025-07-30T03:20:15Z)
• Machine Learning with Privacy for Protected Attributes [56.44253915927481]
  We refine the definition of differential privacy (DP) to create a more general and flexible framework that we call feature differential privacy (FDP)<n>Our definition is simulation-based and allows for both addition/removal and replacement variants of privacy, and can handle arbitrary separation of protected and non-protected features.<n>We apply our framework to various machine learning tasks and show that it can significantly improve the utility of DP-trained models when public features are available.
  arXiv  Detail & Related papers  (2025-06-24T17:53:28Z)
• Differentially Private Random Feature Model [52.468511541184895]
  We produce a differentially private random feature model for privacy-preserving kernel machines. We show that our method preserves privacy and derive a generalization error bound for the method.
  arXiv  Detail & Related papers  (2024-12-06T05:31:08Z)
• To Shuffle or not to Shuffle: Auditing DP-SGD with Shuffling [25.669347036509134]
  We analyze Differentially Private Gradient Descent (DP-SGD) with shuffling. We show that state-of-the-art DP models trained with shuffling appreciably overestimated privacy guarantees (up to 4x) Our work empirically attests to the risk of using shuffling instead of Poisson sub-sampling vis-a-vis the actual privacy leakage of DP-SGD.
  arXiv  Detail & Related papers  (2024-11-15T22:34:28Z)
• Noise Variance Optimization in Differential Privacy: A Game-Theoretic Approach Through Per-Instance Differential Privacy [7.264378254137811]
  Differential privacy (DP) can measure privacy loss by observing the changes in the distribution caused by the inclusion of individuals in the target dataset. DP has been prominent in safeguarding datasets in machine learning in industry giants like Apple and Google. We propose per-instance DP (pDP) as a constraint, measuring privacy loss for each data instance and optimizing noise tailored to individual instances.
  arXiv  Detail & Related papers  (2024-04-24T06:51:16Z)
• How Private are DP-SGD Implementations? [61.19794019914523]
  We show that there can be a substantial gap between the privacy analysis when using the two types of batch sampling. Our result shows that there can be a substantial gap between the privacy analysis when using the two types of batch sampling.
  arXiv  Detail & Related papers  (2024-03-26T13:02:43Z)
• The Symmetric alpha-Stable Privacy Mechanism [0.0]
  We present novel analysis of the Symmetric alpha-Stable (SaS) mechanism. We prove that the mechanism is purely differentially private while remaining closed under convolution.
  arXiv  Detail & Related papers  (2023-11-29T16:34:39Z)
• Additive Logistic Mechanism for Privacy-Preserving Self-Supervised Learning [26.783944764936994]
  We study the privacy risks that are associated with training a neural network's weights with self-supervised learning algorithms. We design a post-training privacy-protection algorithm that adds noise to the fine-tuned weights. We show that the proposed protection algorithm can effectively reduce the attack accuracy to roughly 50%-equivalent to random guessing.
  arXiv  Detail & Related papers  (2022-05-25T01:33:52Z)
• Sampling-Based Robust Control of Autonomous Systems with Non-Gaussian Noise [59.47042225257565]
  We present a novel planning method that does not rely on any explicit representation of the noise distributions. First, we abstract the continuous system into a discrete-state model that captures noise by probabilistic transitions between states. We capture these bounds in the transition probability intervals of a so-called interval Markov decision process (iMDP)
  arXiv  Detail & Related papers  (2021-10-25T06:18:55Z)
• Smoothed Differential Privacy [55.415581832037084]
  Differential privacy (DP) is a widely-accepted and widely-applied notion of privacy based on worst-case analysis. In this paper, we propose a natural extension of DP following the worst average-case idea behind the celebrated smoothed analysis. We prove that any discrete mechanism with sampling procedures is more private than what DP predicts, while many continuous mechanisms with sampling procedures are still non-private under smoothed DP.
  arXiv  Detail & Related papers  (2021-07-04T06:55:45Z)
• Gaussian Processes with Differential Privacy [3.934224774675743]
  We add strong privacy protection to Gaussian processes (GPs) via differential privacy (DP) We achieve this by using sparse GP methodology and publishing a private variational approximation on known inducing points. Our experiments demonstrate that given sufficient amount of data, the method can produce accurate models under strong privacy protection.
  arXiv  Detail & Related papers  (2021-06-01T13:23:16Z)
• On the Practicality of Differential Privacy in Federated Learning by Tuning Iteration Times [51.61278695776151]
  Federated Learning (FL) is well known for its privacy protection when training machine learning models among distributed clients collaboratively. Recent studies have pointed out that the naive FL is susceptible to gradient leakage attacks. Differential Privacy (DP) emerges as a promising countermeasure to defend against gradient leakage attacks.
  arXiv  Detail & Related papers  (2021-01-11T19:43:12Z)
• A One-Pass Private Sketch for Most Machine Learning Tasks [48.17461258268463]
  Differential privacy (DP) is a compelling privacy definition that explains the privacy-utility tradeoff via formal, provable guarantees. We propose a private sketch that supports a multitude of machine learning tasks including regression, classification, density estimation, and more. Our sketch consists of randomized contingency tables that are indexed with locality-sensitive hashing and constructed with an efficient one-pass algorithm.
  arXiv  Detail & Related papers  (2020-06-16T17:47:48Z)
• Differentially Private Federated Learning with Laplacian Smoothing [72.85272874099644]
  Federated learning aims to protect data privacy by collaboratively learning a model without sharing private data among users. An adversary may still be able to infer the private training data by attacking the released model. Differential privacy provides a statistical protection against such attacks at the price of significantly degrading the accuracy or utility of the trained models.
  arXiv  Detail & Related papers  (2020-05-01T04:28:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.