Statistical Inference for Differentially Private Stochastic Gradient Descent

• URL: http://arxiv.org/abs/2507.20560v1
• Date: Mon, 28 Jul 2025 06:45:15 GMT
• Title: Statistical Inference for Differentially Private Stochastic Gradient Descent
• Authors: Xintao Xia, Linjun Zhang, Zhanrui Cai,
• Abstract summary: This paper bridges the gap between existing statistical methods and Differentially Private Gradient Descent (DP-SGD)<n>For the output of DP-SGD, we show that the variance decomposes into statistical, sampling, and privacy-induced components.<n>Two methods are proposed for constructing valid confidence intervals: the plug-in method and the random scaling method.
• Score: 14.360996967498002
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Privacy preservation in machine learning, particularly through Differentially Private Stochastic Gradient Descent (DP-SGD), is critical for sensitive data analysis. However, existing statistical inference methods for SGD predominantly focus on cyclic subsampling, while DP-SGD requires randomized subsampling. This paper first bridges this gap by establishing the asymptotic properties of SGD under the randomized rule and extending these results to DP-SGD. For the output of DP-SGD, we show that the asymptotic variance decomposes into statistical, sampling, and privacy-induced components. Two methods are proposed for constructing valid confidence intervals: the plug-in method and the random scaling method. We also perform extensive numerical analysis, which shows that the proposed confidence intervals achieve nominal coverage rates while maintaining privacy.

Related papers

• Fiducial Matching: Differentially Private Inference for Categorical Data [0.0]
  Inferential statistical inference is still an open area of investigation in a differentially private (DP) setting.<n>We propose a simulation-based matching approach, solved through tools from the fiducial framework.<n>We focus on the analysis of categorical (nominal) data that is common in national surveys.
  arXiv  Detail & Related papers  (2025-07-15T21:56:15Z)
• Second-Order Convergence in Private Stochastic Non-Convex Optimization [28.00987194971941]
  We investigate the problem of finding second-order stationary points (SOS) in differentially private (DP) non-dimensional identification optimization.<n>Existing methods suffer from inaccurate convergence error due to gradient variance in the saddle point escape analysis.<n>We develop a new DP algorithm that rectifies the convergence error reported in prior work.
  arXiv  Detail & Related papers  (2025-05-21T15:25:23Z)
• Private Statistical Estimation via Truncation [2.3910125679710665]
  We introduce a novel framework for differentially private statistical estimation via data truncation, addressing a key challenge in DP estimation when the data support is unbounded.<n>By leveraging techniques from truncated statistics, we develop computationally efficient DP estimators for exponential family distributions.
  arXiv  Detail & Related papers  (2025-05-18T20:38:38Z)
• On the Convergence of DP-SGD with Adaptive Clipping [56.24689348875711]
  Gradient Descent with gradient clipping is a powerful technique for enabling differentially private optimization.<n>This paper provides the first comprehensive convergence analysis of SGD with quantile clipping (QC-SGD)<n>We show how QC-SGD suffers from a bias problem similar to constant-threshold clipped SGD but can be mitigated through a carefully designed quantile and step size schedule.
  arXiv  Detail & Related papers  (2024-12-27T20:29:47Z)
• Scalable DP-SGD: Shuffling vs. Poisson Subsampling [61.19794019914523]
  We provide new lower bounds on the privacy guarantee of the multi-epoch Adaptive Linear Queries (ABLQ) mechanism with shuffled batch sampling. We show substantial gaps when compared to Poisson subsampling; prior analysis was limited to a single epoch. We introduce a practical approach to implement Poisson subsampling at scale using massively parallel computation.
  arXiv  Detail & Related papers  (2024-11-06T19:06:16Z)
• How Private are DP-SGD Implementations? [61.19794019914523]
  We show that there can be a substantial gap between the privacy analysis when using the two types of batch sampling. Our result shows that there can be a substantial gap between the privacy analysis when using the two types of batch sampling.
  arXiv  Detail & Related papers  (2024-03-26T13:02:43Z)
• Differentially Private SGD Without Clipping Bias: An Error-Feedback Approach [62.000948039914135]
  Using Differentially Private Gradient Descent with Gradient Clipping (DPSGD-GC) to ensure Differential Privacy (DP) comes at the cost of model performance degradation. We propose a new error-feedback (EF) DP algorithm as an alternative to DPSGD-GC. We establish an algorithm-specific DP analysis for our proposed algorithm, providing privacy guarantees based on R'enyi DP.
  arXiv  Detail & Related papers  (2023-11-24T17:56:44Z)
• Differentially Private SGDA for Minimax Problems [83.57322009102973]
  We prove that gradient descent ascent (SGDA) can achieve optimal utility in terms of weak primal-dual population risk. This is the first-ever-known result for non-smoothly-strongly-concave setting.
  arXiv  Detail & Related papers  (2022-01-22T13:05:39Z)
• Improving Differentially Private SGD via Randomly Sparsified Gradients [31.295035726077366]
  Differentially private gradient observation (DP-SGD) has been widely adopted in deep learning to provide rigorously defined privacy bound compression. We propose an and utilize RS to strengthen communication cost and strengthen privacy bound compression.
  arXiv  Detail & Related papers  (2021-12-01T21:43:34Z)
• Differentially Private Federated Learning with Laplacian Smoothing [72.85272874099644]
  Federated learning aims to protect data privacy by collaboratively learning a model without sharing private data among users. An adversary may still be able to infer the private training data by attacking the released model. Differential privacy provides a statistical protection against such attacks at the price of significantly degrading the accuracy or utility of the trained models.
  arXiv  Detail & Related papers  (2020-05-01T04:28:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.