The Many Faces of 1-Lipschitz Neural Networks

• URL: http://arxiv.org/abs/2104.05097v2
• Date: Tue, 13 Apr 2021 10:15:02 GMT
• Title: The Many Faces of 1-Lipschitz Neural Networks
• Authors: Louis B\'ethune, Alberto Gonz\'alez-Sanz, Franck Mamalet, Mathieu Serrurier
• Abstract summary: We show that 1-Lipschitz neural network can fit arbitrarily difficult frontier making them as expressive as classical ones. We also study the link between classification with 1-Lipschitz network and optimal transport thanks to regularized versions of Kantorovich-Rubinstein duality theory.
• Score: 1.911678487931003
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Lipschitz constrained models have been used to solve specifics deep learning problems such as the estimation of Wasserstein distance for GAN, or the training of neural networks robust to adversarial attacks. Regardless the novel and effective algorithms to build such 1-Lipschitz networks, their usage remains marginal, and they are commonly considered as less expressive and less able to fit properly the data than their unconstrained counterpart. The goal of the paper is to demonstrate that, despite being empirically harder to train, 1-Lipschitz neural networks are theoretically better grounded than unconstrained ones when it comes to classification. To achieve that we recall some results about 1-Lipschitz function in the scope of deep learning and we extend and illustrate them to derive general properties for classification. First, we show that 1-Lipschitz neural network can fit arbitrarily difficult frontier making them as expressive as classical ones. When minimizing the log loss, we prove that the optimization problem under Lipschitz constraint is well posed and have a minimum, whereas regular neural networks can diverge even on remarkably simple situations. Then, we study the link between classification with 1-Lipschitz network and optimal transport thanks to regularized versions of Kantorovich-Rubinstein duality theory. Last, we derive preliminary bounds on their VC dimension.

Related papers

• LinSATNet: The Positive Linear Satisfiability Neural Networks [116.65291739666303]
  This paper studies how to introduce the popular positive linear satisfiability to neural networks. We propose the first differentiable satisfiability layer based on an extension of the classic Sinkhorn algorithm for jointly encoding multiple sets of marginal distributions.
  arXiv  Detail & Related papers  (2024-07-18T22:05:21Z)
• Upper and lower bounds for the Lipschitz constant of random neural networks [0.0]
  We study upper and lower bounds for the Lipschitz constant of random ReLU neural networks. For shallow neural networks, we characterize the Lipschitz constant up to an absolute numerical constant. For deep networks with fixed depth and sufficiently large width, our established upper bound is larger than the lower bound by a factor that is logarithmic in the width.
  arXiv  Detail & Related papers  (2023-11-02T16:03:26Z)
• A Unified Algebraic Perspective on Lipschitz Neural Networks [88.14073994459586]
  This paper introduces a novel perspective unifying various types of 1-Lipschitz neural networks. We show that many existing techniques can be derived and generalized via finding analytical solutions of a common semidefinite programming (SDP) condition. Our approach, called SDP-based Lipschitz Layers (SLL), allows us to design non-trivial yet efficient generalization of convex potential layers.
  arXiv  Detail & Related papers  (2023-03-06T14:31:09Z)
• Improved techniques for deterministic l2 robustness [63.34032156196848]
  Training convolutional neural networks (CNNs) with a strict 1-Lipschitz constraint under the $l_2$ norm is useful for adversarial robustness, interpretable gradients and stable training. We introduce a procedure to certify robustness of 1-Lipschitz CNNs by replacing the last linear layer with a 1-hidden layer. We significantly advance the state-of-the-art for standard and provable robust accuracies on CIFAR-10 and CIFAR-100.
  arXiv  Detail & Related papers  (2022-11-15T19:10:12Z)
• Improving Lipschitz-Constrained Neural Networks by Learning Activation Functions [14.378778606939665]
  Lipschitz-constrained neural networks have several advantages over unconstrained ones and can be applied to a variety of problems. We show that neural networks with learnable 1-Lipschitz linear splines are known to be more expressive. Our numerical experiments show that our trained networks compare favorably with existing 1-Lipschitz neural architectures.
  arXiv  Detail & Related papers  (2022-10-28T15:56:55Z)
• Rethinking Lipschitz Neural Networks for Certified L-infinity Robustness [33.72713778392896]
  We study certified $ell_infty$ from a novel perspective of representing Boolean functions. We develop a unified Lipschitz network that generalizes prior works, and design a practical version that can be efficiently trained.
  arXiv  Detail & Related papers  (2022-10-04T17:55:27Z)
• Training Certifiably Robust Neural Networks with Efficient Local Lipschitz Bounds [99.23098204458336]
  Certified robustness is a desirable property for deep neural networks in safety-critical applications. We show that our method consistently outperforms state-of-the-art methods on MNIST and TinyNet datasets.
  arXiv  Detail & Related papers  (2021-11-02T06:44:10Z)
• Efficient Proximal Mapping of the 1-path-norm of Shallow Networks [47.20962674178505]
  We show two new important properties of the 1-path-norm neural networks. First, despite its non-smoothness and non-accuracy it allows a closed proximal operator to be efficiently computed. Second, when the activation functions are differentiable, it provides an upper bound on the Lipschitz constant.
  arXiv  Detail & Related papers  (2020-07-02T10:34:06Z)
• On Lipschitz Regularization of Convolutional Layers using Toeplitz Matrix Theory [77.18089185140767]
  Lipschitz regularity is established as a key property of modern deep learning. computing the exact value of the Lipschitz constant of a neural network is known to be NP-hard. We introduce a new upper bound for convolutional layers that is both tight and easy to compute.
  arXiv  Detail & Related papers  (2020-06-15T13:23:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.