PLRV-O: Advancing Differentially Private Deep Learning via Privacy Loss Random Variable Optimization

• URL: http://arxiv.org/abs/2509.06264v1
• Date: Mon, 08 Sep 2025 01:06:45 GMT
• Title: PLRV-O: Advancing Differentially Private Deep Learning via Privacy Loss Random Variable Optimization
• Abstract summary: We introduce PLRV-O, a framework that defines a broad search space of parameterized DP-SGD noise distributions.<n>We show that PLRV-O substantially improves utility under strict privacy constraints.
• Score: 16.48480873870065
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Differentially Private Stochastic Gradient Descent (DP-SGD) is a standard method for enforcing privacy in deep learning, typically using the Gaussian mechanism to perturb gradient updates. However, conventional mechanisms such as Gaussian and Laplacian noise are parameterized only by variance or scale. This single degree of freedom ties the magnitude of noise directly to both privacy loss and utility degradation, preventing independent control of these two factors. The problem becomes more pronounced when the number of composition rounds T and batch size B vary across tasks, as these variations induce task-dependent shifts in the privacy-utility trade-off, where small changes in noise parameters can disproportionately affect model accuracy. To address this limitation, we introduce PLRV-O, a framework that defines a broad search space of parameterized DP-SGD noise distributions, where privacy loss moments are tightly characterized yet can be optimized more independently with respect to utility loss. This formulation enables systematic adaptation of noise to task-specific requirements, including (i) model size, (ii) training duration, (iii) batch sampling strategies, and (iv) clipping thresholds under both training and fine-tuning settings. Empirical results demonstrate that PLRV-O substantially improves utility under strict privacy constraints. On CIFAR-10, a fine-tuned ViT achieves 94.03% accuracy at epsilon approximately 0.5, compared to 83.93% with Gaussian noise. On SST-2, RoBERTa-large reaches 92.20% accuracy at epsilon approximately 0.2, versus 50.25% with Gaussian.

Related papers

• An Adaptive Gradient Clipping and Noise Injection Mechanism for Differentially Private Federated Learning [1.5876034680722233]
  This paper presents DDP-SA-adaptive, an adaptive gradient clipping and noise adding mechanism for differentially private federated learning.<n>Compared with the static DDP-SA baseline, DDP-SA-adaptive reduces the number of communication rounds by 6.81%, total training time by 19.21%, and average per-round training time by 13.33%.<n>It also reduces test loss by 98.74% and increases test R2 by 3.41%, leading to improved model accuracy.
  arXiv  Detail & Related papers  (2026-08-15T10:10:34Z)
• Stabilization of Perturbed Loss Function: Differential Privacy without Gradient Noise [7.010398112169834]
  SPOF is a differentially private training mechanism intended for multi-user environments.<n>It exhibits to environmental noise during training, maintaining stable performance even when user inputs are corrupted.<n>Our results show that SPOF achieves up to 3.5% higher reconstruction accuracy and reduces mean training time by up to 57.2% compared to DP-SGD.
  arXiv  Detail & Related papers  (2025-08-21T12:54:19Z)
• Efficient Differentially Private Fine-Tuning of LLMs via Reinforcement Learning [0.9861588522527782]
  We present RLDP, the first framework to cast DP optimization itself as a closed-loop control problem amenable to modern deep reinforcement learning (RL)<n>Across more than 1,600 experiments on GPT2-small, Llama-1B, Llama-3B, and Mistral-7B, RLDP delivers perplexity reductions of 1.3-3.0.5% and an average 5.6% downstream utility gain.
  arXiv  Detail & Related papers  (2025-07-30T10:46:53Z)
• AdaDPIGU: Differentially Private SGD with Adaptive Clipping and Importance-Based Gradient Updates for Deep Neural Networks [1.7265013728931]
  We propose a new differentially private SGD framework with importance-based gradient updates tailored for deep neural networks.<n>AdaDPIGU satisfies $(varepsilon, delta)$-differential privacy and retains convergence guarantees.<n>On MNIST, our method achieves a test accuracy of 99.12% under a privacy budget of $epsilon = 8$, nearly matching the non-private model.
  arXiv  Detail & Related papers  (2025-07-09T03:53:03Z)
• $(ε, δ)$-Differentially Private Partial Least Squares Regression [1.8666451604540077]
  We propose an $(epsilon, delta)$-differentially private PLS (edPLS) algorithm to ensure the privacy of the data underlying the model.<n> Experimental results demonstrate that edPLS effectively renders privacy attacks, aimed at recovering unique sources of variability in the training data.
  arXiv  Detail & Related papers  (2024-12-12T10:49:55Z)
• Adaptive Differential Privacy in Federated Learning: A Priority-Based Approach [0.0]
  Federated learning (FL) develops global models without direct access to local datasets. DP offers a framework that gives a privacy guarantee by adding certain amounts of noise to parameters. We propose adaptive noise addition in FL which decides the value of injected noise based on features' relative importance.
  arXiv  Detail & Related papers  (2024-01-04T03:01:15Z)
• Amplitude-Varying Perturbation for Balancing Privacy and Utility in Federated Learning [86.08285033925597]
  This paper presents a new DP perturbation mechanism with a time-varying noise amplitude to protect the privacy of federated learning. We derive an online refinement of the series to prevent FL from premature convergence resulting from excessive perturbation noise. The contribution of the new DP mechanism to the convergence and accuracy of privacy-preserving FL is corroborated, compared to the state-of-the-art Gaussian noise mechanism with a persistent noise amplitude.
  arXiv  Detail & Related papers  (2023-03-07T22:52:40Z)
• Improve Noise Tolerance of Robust Loss via Noise-Awareness [60.34670515595074]
  We propose a meta-learning method which is capable of adaptively learning a hyper parameter prediction function, called Noise-Aware-Robust-Loss-Adjuster (NARL-Adjuster for brevity) Four SOTA robust loss functions are attempted to be integrated with our algorithm, and comprehensive experiments substantiate the general availability and effectiveness of the proposed method in both its noise tolerance and performance.
  arXiv  Detail & Related papers  (2023-01-18T04:54:58Z)
• TAN Without a Burn: Scaling Laws of DP-SGD [70.7364032297978]
  Differentially Private methods for training Deep Neural Networks (DNNs) have progressed recently. We decouple privacy analysis and experimental behavior of noisy training to explore the trade-off with minimal computational requirements. We apply the proposed method on CIFAR-10 and ImageNet and, in particular, strongly improve the state-of-the-art on ImageNet with a +9 points gain in top-1 accuracy.
  arXiv  Detail & Related papers  (2022-10-07T08:44:35Z)
• Adaptive Differentially Private Empirical Risk Minimization [95.04948014513226]
  We propose an adaptive (stochastic) gradient perturbation method for differentially private empirical risk minimization. We prove that the ADP method considerably improves the utility guarantee compared to the standard differentially private method in which vanilla random noise is added.
  arXiv  Detail & Related papers  (2021-10-14T15:02:20Z)
• Do Not Let Privacy Overbill Utility: Gradient Embedding Perturbation for Private Learning [74.73901662374921]
  A differentially private model degrades the utility drastically when the model comprises a large number of trainable parameters. We propose an algorithm emphGradient Embedding Perturbation (GEP) towards training differentially private deep models with decent accuracy.
  arXiv  Detail & Related papers  (2021-02-25T04:29:58Z)
• RDP-GAN: A R\'enyi-Differential Privacy based Generative Adversarial Network [75.81653258081435]
  Generative adversarial network (GAN) has attracted increasing attention recently owing to its impressive ability to generate realistic samples with high privacy protection. However, when GANs are applied on sensitive or private training examples, such as medical or financial records, it is still probable to divulge individuals' sensitive and private information. We propose a R'enyi-differentially private-GAN (RDP-GAN), which achieves differential privacy (DP) in a GAN by carefully adding random noises on the value of the loss function during training.
  arXiv  Detail & Related papers  (2020-07-04T09:51:02Z)
• Differentially Private Federated Learning with Laplacian Smoothing [72.85272874099644]
  Federated learning aims to protect data privacy by collaboratively learning a model without sharing private data among users. An adversary may still be able to infer the private training data by attacking the released model. Differential privacy provides a statistical protection against such attacks at the price of significantly degrading the accuracy or utility of the trained models.
  arXiv  Detail & Related papers  (2020-05-01T04:28:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.