Some Fundamental Aspects about Lipschitz Continuity of Neural Networks

• URL: http://arxiv.org/abs/2302.10886v4
• Date: Tue, 14 May 2024 18:19:03 GMT
• Title: Some Fundamental Aspects about Lipschitz Continuity of Neural Networks
• Authors: Grigory Khromov, Sidak Pal Singh,
• Abstract summary: Lipschitz continuity is a crucial functional property of any predictive model. We examine and characterise the Lipschitz behaviour of Neural Networks. We show a remarkable fidelity of the lower Lipschitz bound, identify a striking Double Descent trend in both upper and lower bounds to the Lipschitz and explain the intriguing effects of label noise on function smoothness and generalisation.
• Score: 6.576051895863941
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Lipschitz continuity is a crucial functional property of any predictive model, that naturally governs its robustness, generalisation, as well as adversarial vulnerability. Contrary to other works that focus on obtaining tighter bounds and developing different practical strategies to enforce certain Lipschitz properties, we aim to thoroughly examine and characterise the Lipschitz behaviour of Neural Networks. Thus, we carry out an empirical investigation in a range of different settings (namely, architectures, datasets, label noise, and more) by exhausting the limits of the simplest and the most general lower and upper bounds. As a highlight of this investigation, we showcase a remarkable fidelity of the lower Lipschitz bound, identify a striking Double Descent trend in both upper and lower bounds to the Lipschitz and explain the intriguing effects of label noise on function smoothness and generalisation.

Related papers

• On Robust Reinforcement Learning with Lipschitz-Bounded Policy Networks [1.1060425537315086]
  We show that policy networks with smaller Lipschitz bounds are more robust to disturbances, random noise, and targeted adversarial attacks. We find that the widely-used method of spectral normalization is too conservative and severely impacts clean performance.
  arXiv  Detail & Related papers  (2024-05-19T03:27:31Z)
• A provable control of sensitivity of neural networks through a direct parameterization of the overall bi-Lipschitzness [2.3020018305241337]
  We propose a novel framework for bi-Lipschitzness based on convex neural networks and the Legendre-Fenchel duality. Our framework can achieve such a clear and tight control based on convex neural networks and the Legendre-Fenchel duality.
  arXiv  Detail & Related papers  (2024-04-15T14:21:01Z)
• Novel Quadratic Constraints for Extending LipSDP beyond Slope-Restricted Activations [52.031701581294804]
  Lipschitz bounds for neural networks can be computed with upper time preservation guarantees. Our paper bridges the gap and extends Lipschitz beyond slope-restricted activation functions. Our proposed analysis is general and provides a unified approach for estimating $ell$ and $ell_infty$ Lipschitz bounds.
  arXiv  Detail & Related papers  (2024-01-25T09:23:31Z)
• Efficiently Computing Local Lipschitz Constants of Neural Networks via Bound Propagation [79.13041340708395]
  Lipschitz constants are connected to many properties of neural networks, such as robustness, fairness, and generalization. Existing methods for computing Lipschitz constants either produce relatively loose upper bounds or are limited to small networks. We develop an efficient framework for computing the $ell_infty$ local Lipschitz constant of a neural network by tightly upper bounding the norm of Clarke Jacobian.
  arXiv  Detail & Related papers  (2022-10-13T22:23:22Z)
• Rethinking Lipschitz Neural Networks for Certified L-infinity Robustness [33.72713778392896]
  We study certified $ell_infty$ from a novel perspective of representing Boolean functions. We develop a unified Lipschitz network that generalizes prior works, and design a practical version that can be efficiently trained.
  arXiv  Detail & Related papers  (2022-10-04T17:55:27Z)
• Chordal Sparsity for Lipschitz Constant Estimation of Deep Neural Networks [77.82638674792292]
  Lipschitz constants of neural networks allow for guarantees of robustness in image classification, safety in controller design, and generalizability beyond the training data. As calculating Lipschitz constants is NP-hard, techniques for estimating Lipschitz constants must navigate the trade-off between scalability and accuracy. In this work, we significantly push the scalability frontier of a semidefinite programming technique known as LipSDP while achieving zero accuracy loss.
  arXiv  Detail & Related papers  (2022-04-02T11:57:52Z)
• Training Certifiably Robust Neural Networks with Efficient Local Lipschitz Bounds [99.23098204458336]
  Certified robustness is a desirable property for deep neural networks in safety-critical applications. We show that our method consistently outperforms state-of-the-art methods on MNIST and TinyNet datasets.
  arXiv  Detail & Related papers  (2021-11-02T06:44:10Z)
• Lipschitz Bounded Equilibrium Networks [3.2872586139884623]
  This paper introduces new parameterizations of equilibrium neural networks, i.e. networks defined by implicit equations. The new parameterization admits a Lipschitz bound during training via unconstrained optimization. In image classification experiments we show that the Lipschitz bounds are very accurate and improve robustness to adversarial attacks.
  arXiv  Detail & Related papers  (2020-10-05T01:00:40Z)
• Efficient Proximal Mapping of the 1-path-norm of Shallow Networks [47.20962674178505]
  We show two new important properties of the 1-path-norm neural networks. First, despite its non-smoothness and non-accuracy it allows a closed proximal operator to be efficiently computed. Second, when the activation functions are differentiable, it provides an upper bound on the Lipschitz constant.
  arXiv  Detail & Related papers  (2020-07-02T10:34:06Z)
• On Lipschitz Regularization of Convolutional Layers using Toeplitz Matrix Theory [77.18089185140767]
  Lipschitz regularity is established as a key property of modern deep learning. computing the exact value of the Lipschitz constant of a neural network is known to be NP-hard. We introduce a new upper bound for convolutional layers that is both tight and easy to compute.
  arXiv  Detail & Related papers  (2020-06-15T13:23:34Z)
• The Lipschitz Constant of Self-Attention [27.61634862685452]
  Lipschitz constants of neural networks have been explored in various contexts in deep learning. We investigate the Lipschitz constant of self-attention, a non-linear neural network module widely used in sequence modelling.
  arXiv  Detail & Related papers  (2020-06-08T16:08:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.