On the effect of normalization layers on Differentially Private training
of deep Neural networks
- URL: http://arxiv.org/abs/2006.10919v2
- Date: Tue, 7 Dec 2021 22:55:03 GMT
- Title: On the effect of normalization layers on Differentially Private training
of deep Neural networks
- Authors: Ali Davody, David Ifeoluwa Adelani, Thomas Kleinbauer and Dietrich
Klakow
- Abstract summary: We study the effect of normalization layers on the performance of DPSGD.
We propose a novel method for integrating batch normalization with DPSGD without incurring an additional privacy loss.
- Score: 19.26653302753129
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Differentially private stochastic gradient descent (DPSGD) is a variation of
stochastic gradient descent based on the Differential Privacy (DP) paradigm,
which can mitigate privacy threats that arise from the presence of sensitive
information in training data. However, one major drawback of training deep
neural networks with DPSGD is a reduction in the models accuracy. In this
paper, we study the effect of normalization layers on the performance of DPSGD.
We demonstrate that normalization layers significantly impact the utility of
deep neural networks with noisy parameters and should be considered essential
ingredients of training with DPSGD. In particular, we propose a novel method
for integrating batch normalization with DPSGD without incurring an additional
privacy loss. With our approach, we are able to train deeper networks and
achieve a better utility-privacy trade-off.
Related papers
- Enhancing DP-SGD through Non-monotonous Adaptive Scaling Gradient Weight [15.139854970044075]
We introduce Differentially Private Per-sample Adaptive Scaling Clipping (DP-PSASC)
This approach replaces traditional clipping with non-monotonous adaptive gradient scaling.
Our theoretical and empirical analyses confirm that DP-PSASC preserves gradient privacy and delivers superior performance across diverse datasets.
arXiv Detail & Related papers (2024-11-05T12:47:30Z) - Differential Privacy Regularization: Protecting Training Data Through Loss Function Regularization [49.1574468325115]
Training machine learning models based on neural networks requires large datasets, which may contain sensitive information.
Differentially private SGD [DP-SGD] requires the modification of the standard gradient descent [SGD] algorithm for training new models.
A novel regularization strategy is proposed to achieve the same goal in a more efficient manner.
arXiv Detail & Related papers (2024-09-25T17:59:32Z) - Differentially Private SGD Without Clipping Bias: An Error-Feedback Approach [62.000948039914135]
Using Differentially Private Gradient Descent with Gradient Clipping (DPSGD-GC) to ensure Differential Privacy (DP) comes at the cost of model performance degradation.
We propose a new error-feedback (EF) DP algorithm as an alternative to DPSGD-GC.
We establish an algorithm-specific DP analysis for our proposed algorithm, providing privacy guarantees based on R'enyi DP.
arXiv Detail & Related papers (2023-11-24T17:56:44Z) - Sparsity-Preserving Differentially Private Training of Large Embedding
Models [67.29926605156788]
DP-SGD is a training algorithm that combines differential privacy with gradient descent.
Applying DP-SGD naively to embedding models can destroy gradient sparsity, leading to reduced training efficiency.
We present two new algorithms, DP-FEST and DP-AdaFEST, that preserve gradient sparsity during private training of large embedding models.
arXiv Detail & Related papers (2023-11-14T17:59:51Z) - Domain Generalization Guided by Gradient Signal to Noise Ratio of
Parameters [69.24377241408851]
Overfitting to the source domain is a common issue in gradient-based training of deep neural networks.
We propose to base the selection on gradient-signal-to-noise ratio (GSNR) of network's parameters.
arXiv Detail & Related papers (2023-10-11T10:21:34Z) - Differential Privacy Meets Neural Network Pruning [10.77469946354744]
We study the interplay between neural network pruning and differential privacy, through the two modes of parameter updates.
Our experimental results demonstrate how decreasing the parameter space improves differentially private training.
By studying two popular forms of pruning which do not rely on gradients and do not incur an additional privacy loss, we show that random selection performs on par with magnitude-based selection.
arXiv Detail & Related papers (2023-03-08T14:27:35Z) - Differentially Private Generative Adversarial Networks with Model
Inversion [6.651002556438805]
To protect sensitive data in training a Generative Adversarial Network (GAN), the standard approach is to use differentially private (DP) gradient descent method.
We propose Differentially Private Model Inversion (DPMI) method where the private data is first mapped to the latent space via a public generator.
Our approach outperforms the standard DP-GAN method based on Inception Score, Fr'echet Inception Distance, and classification accuracy under the same privacy guarantee.
arXiv Detail & Related papers (2022-01-10T02:26:26Z) - NeuralDP Differentially private neural networks by design [61.675604648670095]
We propose NeuralDP, a technique for privatising activations of some layer within a neural network.
We experimentally demonstrate on two datasets that our method offers substantially improved privacy-utility trade-offs compared to DP-SGD.
arXiv Detail & Related papers (2021-07-30T12:40:19Z) - Differentially private training of neural networks with Langevin
dynamics forcalibrated predictive uncertainty [58.730520380312676]
We show that differentially private gradient descent (DP-SGD) can yield poorly calibrated, overconfident deep learning models.
This represents a serious issue for safety-critical applications, e.g. in medical diagnosis.
arXiv Detail & Related papers (2021-07-09T08:14:45Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.