Approximating Lipschitz continuous functions with GroupSort neural networks

• URL: http://arxiv.org/abs/2006.05254v2
• Date: Mon, 8 Feb 2021 18:17:50 GMT
• Title: Approximating Lipschitz continuous functions with GroupSort neural networks
• Authors: Ugo Tanielian, Maxime Sangnier, Gerard Biau
• Abstract summary: Recent advances in adversarial attacks and Wasserstein GANs have advocated for use of neural networks with restricted Lipschitz constants. We show in particular how these networks can represent any Lipschitz continuous piecewise linear functions. We also prove that they are well-suited for approximating Lipschitz continuous functions and exhibit upper bounds on both the depth and size.
• Score: 3.416170716497814
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advances in adversarial attacks and Wasserstein GANs have advocated for use of neural networks with restricted Lipschitz constants. Motivated by these observations, we study the recently introduced GroupSort neural networks, with constraints on the weights, and make a theoretical step towards a better understanding of their expressive power. We show in particular how these networks can represent any Lipschitz continuous piecewise linear functions. We also prove that they are well-suited for approximating Lipschitz continuous functions and exhibit upper bounds on both the depth and size. To conclude, the efficiency of GroupSort networks compared with more standard ReLU networks is illustrated in a set of synthetic experiments.

Related papers

• Novel Quadratic Constraints for Extending LipSDP beyond Slope-Restricted Activations [52.031701581294804]
  Lipschitz bounds for neural networks can be computed with upper time preservation guarantees. Our paper bridges the gap and extends Lipschitz beyond slope-restricted activation functions. Our proposed analysis is general and provides a unified approach for estimating $ell$ and $ell_infty$ Lipschitz bounds.
  arXiv  Detail & Related papers  (2024-01-25T09:23:31Z)
• Wide Deep Neural Networks with Gaussian Weights are Very Close to Gaussian Processes [1.0878040851638]
  We show that the distance between the network output and the corresponding Gaussian approximation scales inversely with the width of the network, exhibiting faster convergence than the naive suggested by the central limit theorem. We also apply our bounds to obtain theoretical approximations for the exact posterior distribution of the network, when the likelihood is a bounded Lipschitz function of the network output evaluated on a (finite) training set.
  arXiv  Detail & Related papers  (2023-12-18T22:29:40Z)
• Uniform Convergence of Deep Neural Networks with Lipschitz Continuous Activation Functions and Variable Widths [3.0069322256338906]
  We consider deep neural networks with a Lipschitz continuous activation function and with weight matrices of variable widths. In particular, as convolutional neural networks are special deep neural networks with weight matrices of increasing widths, we put forward conditions on the mask sequence. The Lipschitz continuity assumption on the activation functions allows us to include in our theory most of commonly used activation functions in applications.
  arXiv  Detail & Related papers  (2023-06-02T17:07:12Z)
• Efficient Bound of Lipschitz Constant for Convolutional Layers by Gram Iteration [122.51142131506639]
  We introduce a precise, fast, and differentiable upper bound for the spectral norm of convolutional layers using circulant matrix theory. We show through a comprehensive set of experiments that our approach outperforms other state-of-the-art methods in terms of precision, computational cost, and scalability. It proves highly effective for the Lipschitz regularization of convolutional neural networks, with competitive results against concurrent approaches.
  arXiv  Detail & Related papers  (2023-05-25T15:32:21Z)
• A Unified Algebraic Perspective on Lipschitz Neural Networks [88.14073994459586]
  This paper introduces a novel perspective unifying various types of 1-Lipschitz neural networks. We show that many existing techniques can be derived and generalized via finding analytical solutions of a common semidefinite programming (SDP) condition. Our approach, called SDP-based Lipschitz Layers (SLL), allows us to design non-trivial yet efficient generalization of convex potential layers.
  arXiv  Detail & Related papers  (2023-03-06T14:31:09Z)
• Rethinking Lipschitz Neural Networks for Certified L-infinity Robustness [33.72713778392896]
  We study certified $ell_infty$ from a novel perspective of representing Boolean functions. We develop a unified Lipschitz network that generalizes prior works, and design a practical version that can be efficiently trained.
  arXiv  Detail & Related papers  (2022-10-04T17:55:27Z)
• Robust Implicit Networks via Non-Euclidean Contractions [63.91638306025768]
  Implicit neural networks show improved accuracy and significant reduction in memory consumption. They can suffer from ill-posedness and convergence instability. This paper provides a new framework to design well-posed and robust implicit neural networks.
  arXiv  Detail & Related papers  (2021-06-06T18:05:02Z)
• Analytical bounds on the local Lipschitz constants of ReLU networks [0.0]
  We do so by deriving Lipschitz constants and bounds for ReLU, affine-ReLU, and max pooling functions. Our method produces the largest known bounds on minimum adversarial perturbations for large networks such as AlexNet and VGG-16.
  arXiv  Detail & Related papers  (2021-04-29T21:57:47Z)
• Lipschitz Bounded Equilibrium Networks [3.2872586139884623]
  This paper introduces new parameterizations of equilibrium neural networks, i.e. networks defined by implicit equations. The new parameterization admits a Lipschitz bound during training via unconstrained optimization. In image classification experiments we show that the Lipschitz bounds are very accurate and improve robustness to adversarial attacks.
  arXiv  Detail & Related papers  (2020-10-05T01:00:40Z)
• On Lipschitz Regularization of Convolutional Layers using Toeplitz Matrix Theory [77.18089185140767]
  Lipschitz regularity is established as a key property of modern deep learning. computing the exact value of the Lipschitz constant of a neural network is known to be NP-hard. We introduce a new upper bound for convolutional layers that is both tight and easy to compute.
  arXiv  Detail & Related papers  (2020-06-15T13:23:34Z)
• Lipschitz constant estimation of Neural Networks via sparse polynomial optimization [47.596834444042685]
  LiPopt is a framework for computing increasingly tighter upper bounds on the Lipschitz constant of neural networks. We show how to use the sparse connectivity of a network, to significantly reduce the complexity. We conduct experiments on networks with random weights as well as networks trained on MNIST.
  arXiv  Detail & Related papers  (2020-04-18T18:55:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.