DP-SGD Without Clipping: The Lipschitz Neural Network Way
- URL: http://arxiv.org/abs/2305.16202v2
- Date: Thu, 22 Feb 2024 14:59:51 GMT
- Title: DP-SGD Without Clipping: The Lipschitz Neural Network Way
- Authors: Louis Bethune, Thomas Massena, Thibaut Boissin, Yannick Prudent,
Corentin Friedrich, Franck Mamalet, Aurelien Bellet, Mathieu Serrurier, David
Vigouroux
- Abstract summary: State-of-the-art approaches for training Differentially Private (DP) Deep Neural Networks (DNN)
By bounding the Lipschitz constant of each layer with respect to its parameters, we prove that we can train these networks with privacy guarantees.
Our analysis not only allows the computation of the aforementioned sensitivities at scale, but also provides guidance on how to maximize the gradient-to-noise ratio for fixed privacy guarantees.
- Score: 5.922390405022253
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: State-of-the-art approaches for training Differentially Private (DP) Deep
Neural Networks (DNN) face difficulties to estimate tight bounds on the
sensitivity of the network's layers, and instead rely on a process of
per-sample gradient clipping. This clipping process not only biases the
direction of gradients but also proves costly both in memory consumption and in
computation. To provide sensitivity bounds and bypass the drawbacks of the
clipping process, we propose to rely on Lipschitz constrained networks. Our
theoretical analysis reveals an unexplored link between the Lipschitz constant
with respect to their input and the one with respect to their parameters. By
bounding the Lipschitz constant of each layer with respect to its parameters,
we prove that we can train these networks with privacy guarantees. Our analysis
not only allows the computation of the aforementioned sensitivities at scale,
but also provides guidance on how to maximize the gradient-to-noise ratio for
fixed privacy guarantees. The code has been released as a Python package
available at https://github.com/Algue-Rythme/lip-dp
Related papers
- LipKernel: Lipschitz-Bounded Convolutional Neural Networks via Dissipative Layers [0.0468732641979009]
We propose a layer-wise parameterization for convolutional neural networks (CNNs) that includes built-in robustness guarantees.
Our method Lip Kernel directly parameterizes dissipative convolution kernels using a 2-D Roesser-type state space model.
We show that the run-time using our method is orders of magnitude faster than state-of-the-art Lipschitz-bounded networks.
arXiv Detail & Related papers (2024-10-29T17:20:14Z) - Scalable Lipschitz Estimation for CNNs [3.8125535078871127]
Estimating the Lipschitz constant of deep neural networks is of growing interest.
CNNs underpin much of the recent success in computer vision related applications.
We propose a novel method to accelerate Lipschitz constant estimation for CNNs.
arXiv Detail & Related papers (2024-03-27T14:28:44Z) - Differentially Private SGD Without Clipping Bias: An Error-Feedback Approach [62.000948039914135]
Using Differentially Private Gradient Descent with Gradient Clipping (DPSGD-GC) to ensure Differential Privacy (DP) comes at the cost of model performance degradation.
We propose a new error-feedback (EF) DP algorithm as an alternative to DPSGD-GC.
We establish an algorithm-specific DP analysis for our proposed algorithm, providing privacy guarantees based on R'enyi DP.
arXiv Detail & Related papers (2023-11-24T17:56:44Z) - Direct Parameterization of Lipschitz-Bounded Deep Networks [3.883460584034766]
This paper introduces a new parameterization of deep neural networks (both fully-connected and convolutional) with guaranteed $ell2$ Lipschitz bounds.
The Lipschitz guarantees are equivalent to the tightest-known bounds based on certification via a semidefinite program (SDP)
We provide a direct'' parameterization, i.e., a smooth mapping from $mathbb RN$ onto the set of weights satisfying the SDP-based bound.
arXiv Detail & Related papers (2023-01-27T04:06:31Z) - Efficiently Computing Local Lipschitz Constants of Neural Networks via
Bound Propagation [79.13041340708395]
Lipschitz constants are connected to many properties of neural networks, such as robustness, fairness, and generalization.
Existing methods for computing Lipschitz constants either produce relatively loose upper bounds or are limited to small networks.
We develop an efficient framework for computing the $ell_infty$ local Lipschitz constant of a neural network by tightly upper bounding the norm of Clarke Jacobian.
arXiv Detail & Related papers (2022-10-13T22:23:22Z) - Chordal Sparsity for Lipschitz Constant Estimation of Deep Neural
Networks [77.82638674792292]
Lipschitz constants of neural networks allow for guarantees of robustness in image classification, safety in controller design, and generalizability beyond the training data.
As calculating Lipschitz constants is NP-hard, techniques for estimating Lipschitz constants must navigate the trade-off between scalability and accuracy.
In this work, we significantly push the scalability frontier of a semidefinite programming technique known as LipSDP while achieving zero accuracy loss.
arXiv Detail & Related papers (2022-04-02T11:57:52Z) - Training Certifiably Robust Neural Networks with Efficient Local
Lipschitz Bounds [99.23098204458336]
Certified robustness is a desirable property for deep neural networks in safety-critical applications.
We show that our method consistently outperforms state-of-the-art methods on MNIST and TinyNet datasets.
arXiv Detail & Related papers (2021-11-02T06:44:10Z) - Understanding Gradient Clipping in Private SGD: A Geometric Perspective [68.61254575987013]
Deep learning models are increasingly popular in many machine learning applications where the training data may contain sensitive information.
Many learning systems now incorporate differential privacy by training their models with (differentially) private SGD.
A key step in each private SGD update is gradient clipping that shrinks the gradient of an individual example whenever its L2 norm exceeds some threshold.
arXiv Detail & Related papers (2020-06-27T19:08:12Z) - On Lipschitz Regularization of Convolutional Layers using Toeplitz
Matrix Theory [77.18089185140767]
Lipschitz regularity is established as a key property of modern deep learning.
computing the exact value of the Lipschitz constant of a neural network is known to be NP-hard.
We introduce a new upper bound for convolutional layers that is both tight and easy to compute.
arXiv Detail & Related papers (2020-06-15T13:23:34Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.