Balancing Utility and Privacy: Dynamically Private SGD with Random Projection

• URL: http://arxiv.org/abs/2509.09485v2
• Date: Fri, 12 Sep 2025 01:27:15 GMT
• Title: Balancing Utility and Privacy: Dynamically Private SGD with Random Projection
• Authors: Zhanhong Jiang, Md Zahid Hasan, Nastaran Saadati, Aditya Balu, Chao Liu, Soumik Sarkar,
• Abstract summary: We introduce the Dynamically Differentially Private Projected SGD (D2P2-SGD)<n>We show that D2P2-SGD remarkably enhances accuracy while maintaining privacy.
• Score: 12.562807052680833
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Stochastic optimization is a pivotal enabler in modern machine learning, producing effective models for various tasks. However, several existing works have shown that model parameters and gradient information are susceptible to privacy leakage. Although Differentially Private SGD (DPSGD) addresses privacy concerns, its static noise mechanism impacts the error bounds for model performance. Additionally, with the exponential increase in model parameters, efficient learning of these models using stochastic optimizers has become more challenging. To address these concerns, we introduce the Dynamically Differentially Private Projected SGD (D2P2-SGD) optimizer. In D2P2-SGD, we combine two important ideas: (i) dynamic differential privacy (DDP) with automatic gradient clipping and (ii) random projection with SGD, allowing dynamic adjustment of the tradeoff between utility and privacy of the model. It exhibits provably sub-linear convergence rates across different objective functions, matching the best available rate. The theoretical analysis further suggests that DDP leads to better utility at the cost of privacy, while random projection enables more efficient model learning. Extensive experiments across diverse datasets show that D2P2-SGD remarkably enhances accuracy while maintaining privacy. Our code is available here.

Related papers

• Evaluation of Differential Privacy Mechanisms on Federated Learning [0.0]
  Federated learning is distributed across several clients without disclosing raw data.<n> Differential Privacy (DP) is a technique to protect sensitive data by adding noise to model updates.<n>This work implements DP methods using Laplace and Gaussian mechanisms with an adaptive privacy budget.
  arXiv  Detail & Related papers  (2025-10-09T11:32:36Z)
• Differentially Private Random Feature Model [47.35176457481132]
  We produce a differentially private random feature model for privacy-preserving kernel machines.<n>We show that our method preserves privacy and derive a generalization error bound for the method.
  arXiv  Detail & Related papers  (2024-12-06T05:31:08Z)
• Differentially Private Fine-Tuning of Diffusion Models [22.454127503937883]
  The integration of Differential Privacy with diffusion models (DMs) presents a promising yet challenging frontier. Recent developments in this field have highlighted the potential for generating high-quality synthetic data by pre-training on public data. We propose a strategy optimized for private diffusion models, which minimizes the number of trainable parameters to enhance the privacy-utility trade-off.
  arXiv  Detail & Related papers  (2024-06-03T14:18:04Z)
• Sparsity-Preserving Differentially Private Training of Large Embedding Models [67.29926605156788]
  DP-SGD is a training algorithm that combines differential privacy with gradient descent. Applying DP-SGD naively to embedding models can destroy gradient sparsity, leading to reduced training efficiency. We present two new algorithms, DP-FEST and DP-AdaFEST, that preserve gradient sparsity during private training of large embedding models.
  arXiv  Detail & Related papers  (2023-11-14T17:59:51Z)
• Graphical vs. Deep Generative Models: Measuring the Impact of Differentially Private Mechanisms and Budgets on Utility [18.213030598476198]
  We compare graphical and deep generative models, focusing on the key factors contributing to how privacy budgets are spent. We find that graphical models distribute privacy budgets horizontally and thus cannot handle relatively wide datasets for a fixed training time. Deep generative models spend their budgets per iteration, so their behavior is less predictable with varying dataset dimensions.
  arXiv  Detail & Related papers  (2023-05-18T14:14:42Z)
• Exploring Machine Learning Privacy/Utility trade-off from a hyperparameters Lens [10.727571921061024]
  Differentially Private Descent Gradient (DPSGD) is the state-of-the-art method to train privacy-preserving models. With a drop-in replacement of the activation function, we achieve new state-of-the-art accuracy.
  arXiv  Detail & Related papers  (2023-03-03T09:59:42Z)
• DPIS: An Enhanced Mechanism for Differentially Private SGD with Importance Sampling [23.8561225168394]
  differential privacy (DP) has become a well-accepted standard for privacy protection, and deep neural networks (DNN) have been immensely successful in machine learning. A classic mechanism for this purpose is DP-SGD, which is a differentially private version of the gradient descent (SGD) commonly used for training. We propose DPIS, a novel mechanism for differentially private SGD training that can be used as a drop-in replacement of the core of DP-SGD.
  arXiv  Detail & Related papers  (2022-10-18T07:03:14Z)
• Large Scale Transfer Learning for Differentially Private Image Classification [51.10365553035979]
  Differential Privacy (DP) provides a formal framework for training machine learning models with individual example level privacy. Private training using DP-SGD protects against leakage by injecting noise into individual example gradients. While this result is quite appealing, the computational cost of training large-scale models with DP-SGD is substantially higher than non-private training.
  arXiv  Detail & Related papers  (2022-05-06T01:22:20Z)
• Just Fine-tune Twice: Selective Differential Privacy for Large Language Models [69.66654761324702]
  We propose a simple yet effective just-fine-tune-twice privacy mechanism to achieve SDP for large Transformer-based language models. Experiments show that our models achieve strong performance while staying robust to the canary insertion attack.
  arXiv  Detail & Related papers  (2022-04-15T22:36:55Z)
• Large Language Models Can Be Strong Differentially Private Learners [70.0317718115406]
  Differentially Private (DP) learning has seen limited success for building large deep learning models of text. We show that this performance drop can be mitigated with the use of large pretrained models. We propose a memory saving technique that allows clipping in DP-SGD to run without instantiating per-example gradients.
  arXiv  Detail & Related papers  (2021-10-12T01:45:27Z)
• Differentially Private Federated Learning with Laplacian Smoothing [72.85272874099644]
  Federated learning aims to protect data privacy by collaboratively learning a model without sharing private data among users. An adversary may still be able to infer the private training data by attacking the released model. Differential privacy provides a statistical protection against such attacks at the price of significantly degrading the accuracy or utility of the trained models.
  arXiv  Detail & Related papers  (2020-05-01T04:28:38Z)
• Learnable Bernoulli Dropout for Bayesian Deep Learning [53.79615543862426]
  Learnable Bernoulli dropout (LBD) is a new model-agnostic dropout scheme that considers the dropout rates as parameters jointly optimized with other model parameters. LBD leads to improved accuracy and uncertainty estimates in image classification and semantic segmentation.
  arXiv  Detail & Related papers  (2020-02-12T18:57:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.