An Improved Privacy and Utility Analysis of Differentially Private SGD with Bounded Domain and Smooth Losses

• URL: http://arxiv.org/abs/2502.17772v2
• Date: Fri, 28 Feb 2025 05:45:00 GMT
• Title: An Improved Privacy and Utility Analysis of Differentially Private SGD with Bounded Domain and Smooth Losses
• Authors: Hao Liang, Wanrong Zhang, Xinlei He, Kaishun Wu, Hong Xing,
• Abstract summary: We provide rigorous privacy and utility characterization for Differentially Private Gradient Descent.<n>We show that for DPSGD with a bounded domain, the privacy loss can still converge without the convexity assumption.
• Score: 26.98562734120675
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Differentially Private Stochastic Gradient Descent (DPSGD) is widely used to protect sensitive data during the training of machine learning models, but its privacy guarantees often come at the cost of model performance, largely due to the inherent challenge of accurately quantifying privacy loss. While recent efforts have strengthened privacy guarantees by focusing solely on the final output and bounded domain cases, they still impose restrictive assumptions, such as convexity and other parameter limitations, and often lack a thorough analysis of utility. In this paper, we provide rigorous privacy and utility characterization for DPSGD for smooth loss functions in both bounded and unbounded domains. We track the privacy loss over multiple iterations by exploiting the noisy smooth-reduction property and establish the utility analysis by leveraging the projection's non-expansiveness and clipped SGD properties. In particular, we show that for DPSGD with a bounded domain, (i) the privacy loss can still converge without the convexity assumption, and (ii) a smaller bounded diameter can improve both privacy and utility simultaneously under certain conditions. Numerical results validate our results.

Related papers

• Linear-Time User-Level DP-SCO via Robust Statistics [55.350093142673316]
  User-level differentially private convex optimization (DP-SCO) has garnered significant attention due to the importance of safeguarding user privacy in machine learning applications. Current methods, such as those based on differentially private gradient descent (DP-SGD), often struggle with high noise accumulation and suboptimal utility. We introduce a novel linear-time algorithm that leverages robust statistics, specifically the median and trimmed mean, to overcome these challenges.
  arXiv  Detail & Related papers  (2025-02-13T02:05:45Z)
• Meeting Utility Constraints in Differential Privacy: A Privacy-Boosting Approach [7.970280110429423]
  We propose a privacy-boosting framework that is compatible with most noise-adding DP mechanisms. Our framework enhances the likelihood of outputs falling within a preferred subset of the support to meet utility requirements. We show that our framework achieves lower privacy loss than standard DP mechanisms under utility constraints.
  arXiv  Detail & Related papers  (2024-12-13T23:34:30Z)
• The Last Iterate Advantage: Empirical Auditing and Principled Heuristic Analysis of Differentially Private SGD [46.71175773861434]
  We propose a simple privacy analysis of noisy clipped gradient descent (DP-SGD) We show experimentally that our is predictive of the outcome of privacy auditing applied to various training procedures. We also empirically support our and show existing privacy auditing attacks are bounded by our analysis in both vision and language tasks.
  arXiv  Detail & Related papers  (2024-10-08T16:51:10Z)
• Deep Learning with Data Privacy via Residual Perturbation [13.437435287800668]
  Several celebrated privacy notions have been established and used for privacy-preserving deep learning (DL) In this paper, we propose a differential equation-based residual perturbation for privacy-preserving DL. We show that residual perturbation is efficient and outperforms the state-of-the-art differentially private descent.
  arXiv  Detail & Related papers  (2024-08-11T08:26:43Z)
• Private Optimal Inventory Policy Learning for Feature-based Newsvendor with Unknown Demand [13.594765018457904]
  This paper introduces a novel approach to estimate a privacy-preserving optimal inventory policy within the f-differential privacy framework. We develop a clipped noisy gradient descent algorithm based on convolution smoothing for optimal inventory estimation. Our numerical experiments demonstrate that the proposed new method can achieve desirable privacy protection with a marginal increase in cost.
  arXiv  Detail & Related papers  (2024-04-23T19:15:43Z)
• Differentially Private SGD Without Clipping Bias: An Error-Feedback Approach [62.000948039914135]
  Using Differentially Private Gradient Descent with Gradient Clipping (DPSGD-GC) to ensure Differential Privacy (DP) comes at the cost of model performance degradation. We propose a new error-feedback (EF) DP algorithm as an alternative to DPSGD-GC. We establish an algorithm-specific DP analysis for our proposed algorithm, providing privacy guarantees based on R'enyi DP.
  arXiv  Detail & Related papers  (2023-11-24T17:56:44Z)
• Privacy Loss of Noisy Stochastic Gradient Descent Might Converge Even for Non-Convex Losses [4.68299658663016]
  The Noisy-SGD algorithm is widely used for privately training machine learning models. Recent findings have shown that if the internal state remains hidden, then the privacy loss might remain bounded. We address this problem for DP-SGD, a popular variant of Noisy-SGD that incorporates gradient clipping to limit the impact of individual samples on the training process.
  arXiv  Detail & Related papers  (2023-05-17T02:25:56Z)
• How Do Input Attributes Impact the Privacy Loss in Differential Privacy? [55.492422758737575]
  We study the connection between the per-subject norm in DP neural networks and individual privacy loss. We introduce a novel metric termed the Privacy Loss-Input Susceptibility (PLIS) which allows one to apportion the subject's privacy loss to their input attributes.
  arXiv  Detail & Related papers  (2022-11-18T11:39:03Z)
• On the Statistical Complexity of Estimation and Testing under Privacy Constraints [17.04261371990489]
  We show how to characterize the power of a statistical test under differential privacy in a plug-and-play fashion. We show that maintaining privacy results in a noticeable reduction in performance only when the level of privacy protection is very high. Finally, we demonstrate that the DP-SGLD algorithm, a private convex solver, can be employed for maximum likelihood estimation with a high degree of confidence.
  arXiv  Detail & Related papers  (2022-10-05T12:55:53Z)
• Private Domain Adaptation from a Public Source [48.83724068578305]
  We design differentially private discrepancy-based algorithms for adaptation from a source domain with public labeled data to a target domain with unlabeled private data. Our solutions are based on private variants of Frank-Wolfe and Mirror-Descent algorithms.
  arXiv  Detail & Related papers  (2022-08-12T06:52:55Z)
• Do Not Let Privacy Overbill Utility: Gradient Embedding Perturbation for Private Learning [74.73901662374921]
  A differentially private model degrades the utility drastically when the model comprises a large number of trainable parameters. We propose an algorithm emphGradient Embedding Perturbation (GEP) towards training differentially private deep models with decent accuracy.
  arXiv  Detail & Related papers  (2021-02-25T04:29:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.