Less Random, More Private: What is the Optimal Subsampling Scheme for DP-SGD?

• URL: http://arxiv.org/abs/2605.07072v1
• Date: Fri, 08 May 2026 00:47:11 GMT
• Title: Less Random, More Private: What is the Optimal Subsampling Scheme for DP-SGD?
• Abstract summary: We show that Balanced It Subsampling (BIS) achieves stronger privacy amplification than Poisson subsampling and is optimal at both extremes of the noise spectrum.<n>We introduce a practical near-exact Monte Carlo accountant for BIS, which removes analytical slack of existing and composition-based PLD analyses.
• Score: 14.158845925610438
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Poisson subsampling is the default sampling scheme in differentially private machine learning, largely because its unstructured randomness yields tractable privacy amplification analyses. Yet this same randomness introduces substantial participation variance: each sample appears in very different numbers of training iterations. In this work, we show that this variance is not merely a practical artifact to be tolerated, but a fundamental source of suboptimal privacy amplification. We prove that Balanced Iteration Subsampling (BIS), a structured scheme in which each sample participates in exactly a fixed number of iterations, achieves stronger privacy amplification than Poisson subsampling and is optimal at both extremes of the noise spectrum ($σ\to 0$ and $σ\to \infty$). Our analysis reveals that the privacy-noise tradeoff is governed not by maximizing randomness, but by eliminating participation variance while preserving uniform marginal participation across iterations. To translate this asymptotic theory into finite-noise guarantees, we introduce a practical near-exact Monte Carlo accountant for BIS, which removes the analytical slack of existing RDP and composition-based PLD analyses. Evaluations across more than 60 practical DP-SGD configurations show that BIS consistently outperforms Poisson subsampling in the low-noise regimes most relevant for high-utility private training, reducing the required noise multiplier by up to $9.6\%$. These results overturn the common intuition that more sampling randomness necessarily yields stronger privacy amplification: in DP-SGD, structured participation can be both more practical and more private. Our implementation is available at https://github.com/dong-xin-ao-andy/bis-mc-accountant.

Related papers

• Efficient privacy loss accounting for subsampling and random allocation [13.186391867041394]
  We consider the privacy amplification properties of a sampling scheme in which a user's data is used in random steps.<n>We show that the privacy-utility trade-off for random allocation is at least as good as that of Poisson subsampling.
  arXiv  Detail & Related papers  (2026-02-19T11:44:25Z)
• LoRA and Privacy: When Random Projections Help (and When They Don't) [55.65932772290123]
  We introduce the (Wishart) projection mechanism, a randomized map of the form $S mapsto M f(S)$ with $M sim W_d (1/r I_d, r)$ and study its differential privacy properties.<n>For vector-valued queries $f$, we prove non-asymptotic DP guarantees without any additive noise, showing that Wishart randomness alone can suffice.<n>For matrix-valued queries, however, we establish a sharp negative result: in the noise-free setting, the mechanism is not DP, and we demonstrate its vulnerability.
  arXiv  Detail & Related papers  (2026-01-29T13:43:37Z)
• Fundamental Limitations of Favorable Privacy-Utility Guarantees for DP-SGD [7.787109481104569]
  We analyze Differentially Private Gradient Descent (DP-SGD) in the $f$differential privacy framework.<n>We prove that enforcing a small separation imposes a strict lower bound on the noise multiplier $$, which directly limits the achievable utility.<n>Our experiments confirm that the noise levels implied by this bound leads to significant accuracy degradation at realistic training settings.
  arXiv  Detail & Related papers  (2026-01-15T09:50:36Z)
• Improved Sample Complexity for Full Coverage in Compact and Continuous Spaces [0.0]
  We study uniform random sampling on the $d$-dimensional unit hypercube.<n>We derive a sample complexity bound with a logarithmic dependence on the failure probability.<n>Our findings offer a sharper theoretical tool for algorithms that rely on grid-based coverage guarantees.
  arXiv  Detail & Related papers  (2025-11-21T21:06:14Z)
• The Cost of Shuffling in Private Gradient Based Optimization [40.31928071333575]
  We show that data shuffling results in worse empirical excess risk for textitDP-ShuffleG compared to DP-SGD.<n>We propose textitInterleaved-ShuffleG, a hybrid approach that integrates public data samples in private optimization.
  arXiv  Detail & Related papers  (2025-02-05T22:30:00Z)
• Balls-and-Bins Sampling for DP-SGD [56.959167524617456]
  We introduce the Balls-and-Bins sampling for differentially private (DP) optimization methods such as DP-SGD.<n>We show that Balls-and-Bins sampling achieves the "best-of-both" samplers, namely, the implementation of Balls-and-Bins sampling is similar to that of Shuffling.
  arXiv  Detail & Related papers  (2024-12-21T23:09:14Z)
• Scalable DP-SGD: Shuffling vs. Poisson Subsampling [61.19794019914523]
  We provide new lower bounds on the privacy guarantee of the multi-epoch Adaptive Linear Queries (ABLQ) mechanism with shuffled batch sampling. We show substantial gaps when compared to Poisson subsampling; prior analysis was limited to a single epoch. We introduce a practical approach to implement Poisson subsampling at scale using massively parallel computation.
  arXiv  Detail & Related papers  (2024-11-06T19:06:16Z)
• How Private are DP-SGD Implementations? [61.19794019914523]
  We show that there can be a substantial gap between the privacy analysis when using the two types of batch sampling. Our result shows that there can be a substantial gap between the privacy analysis when using the two types of batch sampling.
  arXiv  Detail & Related papers  (2024-03-26T13:02:43Z)
• Differentially-Private Bayes Consistency [70.92545332158217]
  We construct a Bayes consistent learning rule that satisfies differential privacy (DP) We prove that any VC class can be privately learned in a semi-supervised setting with a near-optimal sample complexity.
  arXiv  Detail & Related papers  (2022-12-08T11:57:30Z)
• Smoothed Differential Privacy [55.415581832037084]
  Differential privacy (DP) is a widely-accepted and widely-applied notion of privacy based on worst-case analysis. In this paper, we propose a natural extension of DP following the worst average-case idea behind the celebrated smoothed analysis. We prove that any discrete mechanism with sampling procedures is more private than what DP predicts, while many continuous mechanisms with sampling procedures are still non-private under smoothed DP.
  arXiv  Detail & Related papers  (2021-07-04T06:55:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.