Rényi Differential Privacy for Heavy-Tailed SDEs via Fractional Poincaré Inequalities

• URL: http://arxiv.org/abs/2511.15634v1
• Date: Wed, 19 Nov 2025 17:18:54 GMT
• Title: Rényi Differential Privacy for Heavy-Tailed SDEs via Fractional Poincaré Inequalities
• Authors: Benjamin Dupuis, Mert Gürbüzbalaban, Umut Şimşekli, Jian Wang, Sinan Yildirim, Lingjiong Zhu,
• Abstract summary: Recently, the first DP guarantees for heavy-tailed gradient descent were obtained.<n>Despite casting new light on the differential link between DP and heavy-tailed SGD, these results have a strong dependence on the number of parameters.<n>Our framework is based on new Rényi flow computations and the use of well-established fractional Poincaré inequalities.
• Score: 14.760162530695661
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Characterizing the differential privacy (DP) of learning algorithms has become a major challenge in recent years. In parallel, many studies suggested investigating the behavior of stochastic gradient descent (SGD) with heavy-tailed noise, both as a model for modern deep learning models and to improve their performance. However, most DP bounds focus on light-tailed noise, where satisfactory guarantees have been obtained but the proposed techniques do not directly extend to the heavy-tailed setting. Recently, the first DP guarantees for heavy-tailed SGD were obtained. These results provide $(0,δ)$-DP guarantees without requiring gradient clipping. Despite casting new light on the link between DP and heavy-tailed algorithms, these results have a strong dependence on the number of parameters and cannot be extended to other DP notions like the well-established Rényi differential privacy (RDP). In this work, we propose to address these limitations by deriving the first RDP guarantees for heavy-tailed SDEs, as well as their discretized counterparts. Our framework is based on new Rényi flow computations and the use of well-established fractional Poincaré inequalities. Under the assumption that such inequalities are satisfied, we obtain DP guarantees that have a much weaker dependence on the dimension compared to prior art.

Related papers

• Enhancing DPSGD via Per-Sample Momentum and Low-Pass Filtering [4.9871580445771455]
  Differentially Private Gradient Descent (DPSGD) is widely used to train deep neural networks with formal privacy guarantees.<n>Existing techniques typically address only one of these issues, as reducing DP noise can exacerbate clipping bias and vice-versa.<n>We propose a novel method, emphDP-PMLF, which integrates per-sample momentum with a low-pass filtering strategy to simultaneously mitigate DP noise and clipping bias.
  arXiv  Detail & Related papers  (2025-11-11T23:40:02Z)
• Double Momentum and Error Feedback for Clipping with Fast Rates and Differential Privacy [11.356444450240799]
  existing algorithms do not achieve Strong Differential Privacy (DP) and Optimization guarantees at once.<n>We propose a new method called Clip21-SGD2M based on a novel combination of clipping, heavy-ball momentum, and Error Feedback.
  arXiv  Detail & Related papers  (2025-02-17T11:16:21Z)
• Differentially Private Policy Gradient [48.748194765816955]
  We show that it is possible to find the right trade-off between privacy noise and trust-region size to obtain a performant differentially private policy gradient algorithm.<n>Our results and the complexity of the tasks addressed represent a significant improvement over existing DP algorithms in online RL.
  arXiv  Detail & Related papers  (2025-01-31T12:11:13Z)
• Privacy of SGD under Gaussian or Heavy-Tailed Noise: Guarantees without Gradient Clipping [15.348717323408652]
  We bridge the gap between theoretical and empirical benefits of heavy-tailed noise for optimization and privacy preservation.<n>We show that heavy-tailed noising schemes can be a viable alternative to their light-tailed counterparts.
  arXiv  Detail & Related papers  (2024-03-04T13:53:41Z)
• Differentially Private SGD Without Clipping Bias: An Error-Feedback Approach [62.000948039914135]
  Using Differentially Private Gradient Descent with Gradient Clipping (DPSGD-GC) to ensure Differential Privacy (DP) comes at the cost of model performance degradation. We propose a new error-feedback (EF) DP algorithm as an alternative to DPSGD-GC. We establish an algorithm-specific DP analysis for our proposed algorithm, providing privacy guarantees based on R'enyi DP.
  arXiv  Detail & Related papers  (2023-11-24T17:56:44Z)
• Towards the Flatter Landscape and Better Generalization in Federated Learning under Client-level Differential Privacy [67.33715954653098]
  We propose a novel DPFL algorithm named DP-FedSAM, which leverages gradient perturbation to mitigate the negative impact of DP. Specifically, DP-FedSAM integrates Sharpness Aware of Minimization (SAM) to generate local flatness models with stability and weight robustness. To further reduce the magnitude random noise while achieving better performance, we propose DP-FedSAM-$top_k$ by adopting the local update sparsification technique.
  arXiv  Detail & Related papers  (2023-05-01T15:19:09Z)
• Normalized/Clipped SGD with Perturbation for Differentially Private Non-Convex Optimization [94.06564567766475]
  DP-SGD and DP-NSGD mitigate the risk of large models memorizing sensitive training data. We show that these two algorithms achieve similar best accuracy while DP-NSGD is comparatively easier to tune than DP-SGD.
  arXiv  Detail & Related papers  (2022-06-27T03:45:02Z)
• On Private Online Convex Optimization: Optimal Algorithms in $\ell_p$-Geometry and High Dimensional Contextual Bandits [9.798304879986604]
  We study the Differentially private (DP) convex optimization problem with streaming data sampled from a distribution and arrives sequentially. We also consider the continual release model where parameters related to private information are updated and released upon each new data, often known as the online algorithms. Our algorithm achieves in linear time the optimal excess risk when $1pleq 2$ and the state-of-the-art excess risk meeting the non-private lower ones when $2pleqinfty$.
  arXiv  Detail & Related papers  (2022-06-16T12:09:47Z)
• Differentially Private SGDA for Minimax Problems [83.57322009102973]
  We prove that gradient descent ascent (SGDA) can achieve optimal utility in terms of weak primal-dual population risk. This is the first-ever-known result for non-smoothly-strongly-concave setting.
  arXiv  Detail & Related papers  (2022-01-22T13:05:39Z)
• Smoothed Differential Privacy [55.415581832037084]
  Differential privacy (DP) is a widely-accepted and widely-applied notion of privacy based on worst-case analysis. In this paper, we propose a natural extension of DP following the worst average-case idea behind the celebrated smoothed analysis. We prove that any discrete mechanism with sampling procedures is more private than what DP predicts, while many continuous mechanisms with sampling procedures are still non-private under smoothed DP.
  arXiv  Detail & Related papers  (2021-07-04T06:55:45Z)
• Three Variants of Differential Privacy: Lossless Conversion and Applications [13.057076084452016]
  We consider three different variants of differential privacy (DP), namely approximate DP, R'enyi RDP, and hypothesis test. In the first part, we develop a machinery for relating approximate DP to iterations based on the joint range of two $f$-divergences. As an application, we apply our result to the moments framework for characterizing privacy guarantees of noisy gradient descent.
  arXiv  Detail & Related papers  (2020-08-14T18:23:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.