DP-IVON-Gradsq: Differentially Private Squared-Gradient Improved Variational Online Newton

• URL: http://arxiv.org/abs/2607.23649v1
• Date: Sun, 26 Jul 2026 13:32:49 GMT
• Title: DP-IVON-Gradsq: Differentially Private Squared-Gradient Improved Variational Online Newton
• Abstract summary: We investigate differentially private variational Bayesian learning through the Improved Variational Online (IVON) framework.<n>The proposed method constructs its curvature estimate using a noise-corrected squared-gradient estimator.<n>We evaluate DP-IVON-sq on CIFAR-10 against the standard privates DP-SGD and DP-Adam on a range of privacy budgets.
• Score: 7.840001337179043
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Differential privacy provides formal privacy guarantees for training neural networks on sensitive data, while Bayesian deep learning offers a principled framework for uncertainty-aware prediction. Combining these two objectives remains challenging, as privacy noise can interact with the stochasticity introduced by Bayesian posterior sampling. In this work, we investigate differentially private variational Bayesian learning through the Improved Variational Online Newton (IVON) optimizer. We introduce DP-IVON-Gradsq, a private variant of IVON. The proposed method constructs its curvature estimate from the privatized gradient using a noise-corrected squared-gradient estimator, reducing the direct interaction between posterior-sampling noise and privacy noise while preserving the Adam-like computational efficiency of IVON. We evaluate DP-IVON-Gradsq on CIFAR-10 against the standard private optimizers DP-SGD and DP-Adam over a range of privacy budgets. The results show that DP-IVON-Gradsq is competitive under weak-to-moderate privacy constraints, i.e., large-to-moderate values of $\varepsilon$, while degrading under strong privacy. Code is available at https://github.com/NourJamoussi/DP-IVON-Gradsq.git.

Related papers

• Privacy Without Regret: Differentially Private Inference-Time Alignment [8.568849346613602]
  We show that a single intervention-adding noise to reward scores before selection-resolves both.<n>PrivBoN establishes that Gumbel noise at an appropriate scale simultaneously provides $$-differential privacy.<n>PrivITP achieves ex-post $(,)$-DP with a privacy cost independent of the number of responses.
  arXiv  Detail & Related papers  (2026-08-26T18:56:27Z)
• Evaluation of Differential Privacy Mechanisms on Federated Learning [0.0]
  Federated learning is distributed across several clients without disclosing raw data.<n> Differential Privacy (DP) is a technique to protect sensitive data by adding noise to model updates.<n>This work implements DP methods using Laplace and Gaussian mechanisms with an adaptive privacy budget.
  arXiv  Detail & Related papers  (2025-10-09T11:32:36Z)
• Revisiting Privacy-Utility Trade-off for DP Training with Pre-existing Knowledge [40.44144653519249]
  We propose a generic differential privacy framework with heterogeneous noise (DP-Hero)<n>Atop DP-Hero, we instantiate a heterogeneous version of DP-SGD, and further extend it to federated training.<n>We conduct comprehensive experiments to verify and explain the effectiveness of the proposed DP-Hero, showing improved training accuracy compared with state-of-the-art works.
  arXiv  Detail & Related papers  (2024-09-05T08:40:54Z)
• Differentially Private Best-Arm Identification [14.916947598339988]
  Best Arm Identification (BAI) problems are progressively used for data-sensitive applications. Motivated by the data privacy concerns invoked by these applications, we study the problem of BAI with fixed confidence in both the local and central models.
  arXiv  Detail & Related papers  (2024-06-10T16:02:48Z)
• Private Fine-tuning of Large Language Models with Zeroth-order Optimization [51.19403058739522]
  Differentially private gradient descent (DP-SGD) allows models to be trained in a privacy-preserving manner.<n>We introduce DP-ZO, a private fine-tuning framework for large language models by privatizing zeroth order optimization methods.
  arXiv  Detail & Related papers  (2024-01-09T03:53:59Z)
• Blink: Link Local Differential Privacy in Graph Neural Networks via Bayesian Estimation [79.64626707978418]
  We propose using link local differential privacy over decentralized nodes to train graph neural networks. Our approach spends the privacy budget separately on links and degrees of the graph for the server to better denoise the graph topology. Our approach outperforms existing methods in terms of accuracy under varying privacy budgets.
  arXiv  Detail & Related papers  (2023-09-06T17:53:31Z)
• "Private Prediction Strikes Back!'' Private Kernelized Nearest Neighbors with Individual Renyi Filter [31.970442970375153]
  We propose an algorithm named Individualized Nearest Neighbor (Ind-KNN) for private prediction. Ind-KNN is easily updatable over dataset changes and it allows precise control of the R'enyi at an individual user level. Our results show that Ind-KNN consistently improves the accuracy over existing private prediction methods for a wide range of $epsilon$ on four vision and language tasks.
  arXiv  Detail & Related papers  (2023-06-12T19:14:45Z)
• TAN Without a Burn: Scaling Laws of DP-SGD [70.7364032297978]
  Differentially Private methods for training Deep Neural Networks (DNNs) have progressed recently. We decouple privacy analysis and experimental behavior of noisy training to explore the trade-off with minimal computational requirements. We apply the proposed method on CIFAR-10 and ImageNet and, in particular, strongly improve the state-of-the-art on ImageNet with a +9 points gain in top-1 accuracy.
  arXiv  Detail & Related papers  (2022-10-07T08:44:35Z)
• DP$^2$-VAE: Differentially Private Pre-trained Variational Autoencoders [26.658723213776632]
  We propose DP$2$-VAE, a training mechanism for variational autoencoders (VAE) with provable DP guarantees and improved utility via emphpre-training on private data. We conduct extensive experiments on image datasets to illustrate our superiority over baselines under various privacy budgets and evaluation metrics.
  arXiv  Detail & Related papers  (2022-08-05T23:57:34Z)
• Individual Privacy Accounting for Differentially Private Stochastic Gradient Descent [69.14164921515949]
  We characterize privacy guarantees for individual examples when releasing models trained by DP-SGD. We find that most examples enjoy stronger privacy guarantees than the worst-case bound. This implies groups that are underserved in terms of model utility simultaneously experience weaker privacy guarantees.
  arXiv  Detail & Related papers  (2022-06-06T13:49:37Z)
• Don't Generate Me: Training Differentially Private Generative Models with Sinkhorn Divergence [73.14373832423156]
  We propose DP-Sinkhorn, a novel optimal transport-based generative method for learning data distributions from private data with differential privacy. Unlike existing approaches for training differentially private generative models, we do not rely on adversarial objectives.
  arXiv  Detail & Related papers  (2021-11-01T18:10:21Z)
• Smoothed Differential Privacy [55.415581832037084]
  Differential privacy (DP) is a widely-accepted and widely-applied notion of privacy based on worst-case analysis. In this paper, we propose a natural extension of DP following the worst average-case idea behind the celebrated smoothed analysis. We prove that any discrete mechanism with sampling procedures is more private than what DP predicts, while many continuous mechanisms with sampling procedures are still non-private under smoothed DP.
  arXiv  Detail & Related papers  (2021-07-04T06:55:45Z)
• RDP-GAN: A R\'enyi-Differential Privacy based Generative Adversarial Network [75.81653258081435]
  Generative adversarial network (GAN) has attracted increasing attention recently owing to its impressive ability to generate realistic samples with high privacy protection. However, when GANs are applied on sensitive or private training examples, such as medical or financial records, it is still probable to divulge individuals' sensitive and private information. We propose a R'enyi-differentially private-GAN (RDP-GAN), which achieves differential privacy (DP) in a GAN by carefully adding random noises on the value of the loss function during training.
  arXiv  Detail & Related papers  (2020-07-04T09:51:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.