A Convex Parameterization of Robust Recurrent Neural Networks

• URL: http://arxiv.org/abs/2004.05290v2
• Date: Sat, 3 Oct 2020 08:48:04 GMT
• Title: A Convex Parameterization of Robust Recurrent Neural Networks
• Authors: Max Revay, Ruigang Wang, Ian R. Manchester
• Abstract summary: Recurrent neural networks (RNNs) are a class of nonlinear dynamical systems often used to model sequence-to-sequence maps. We formulate convex sets of RNNs with stability and robustness guarantees.
• Score: 3.2872586139884623
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recurrent neural networks (RNNs) are a class of nonlinear dynamical systems often used to model sequence-to-sequence maps. RNNs have excellent expressive power but lack the stability or robustness guarantees that are necessary for many applications. In this paper, we formulate convex sets of RNNs with stability and robustness guarantees. The guarantees are derived using incremental quadratic constraints and can ensure global exponential stability of all solutions, and bounds on incremental $ \ell_2 $ gain (the Lipschitz constant of the learned sequence-to-sequence mapping). Using an implicit model structure, we construct a parametrization of RNNs that is jointly convex in the model parameters and stability certificate. We prove that this model structure includes all previously-proposed convex sets of stable RNNs as special cases, and also includes all stable linear dynamical systems. We illustrate the utility of the proposed model class in the context of non-linear system identification.

Related papers

• Unconditional stability of a recurrent neural circuit implementing divisive normalization [0.0]
  We prove the remarkable property of unconditional local stability for an arbitrary-dimensional ORGaNICs circuit. We show that ORGaNICs can be trained by backpropagation through time without gradient clipping/scaling.
  arXiv  Detail & Related papers  (2024-09-27T17:46:05Z)
• PAC-Bayes Generalisation Bounds for Dynamical Systems Including Stable RNNs [11.338419403452239]
  We derive a PAC-Bayes bound on the generalisation gap for a special class of discrete-time non-linear dynamical systems. The proposed bound converges to zero as the dataset size increases. Unlike other available bounds the derived bound holds for non i.i.d. data (time-series) and it does not grow with the number of steps of the RNN.
  arXiv  Detail & Related papers  (2023-12-15T13:49:29Z)
• Second-order regression models exhibit progressive sharpening to the edge of stability [30.92413051155244]
  We show that for quadratic objectives in two dimensions, a second-order regression model exhibits progressive sharpening towards a value that differs slightly from the edge of stability. In higher dimensions, the model generically shows similar behavior, even without the specific structure of a neural network.
  arXiv  Detail & Related papers  (2022-10-10T17:21:20Z)
• Lipschitz Continuity Retained Binary Neural Network [52.17734681659175]
  We introduce the Lipschitz continuity as the rigorous criteria to define the model robustness for BNN. We then propose to retain the Lipschitz continuity as a regularization term to improve the model robustness. Our experiments prove that our BNN-specific regularization method can effectively strengthen the robustness of BNN.
  arXiv  Detail & Related papers  (2022-07-13T22:55:04Z)
• Controlling the Complexity and Lipschitz Constant improves polynomial nets [55.121200972539114]
  We derive new complexity bounds for the set of Coupled CP-Decomposition (CCP) and Nested Coupled CP-decomposition (NCP) models of Polynomial Nets. We propose a principled regularization scheme that we evaluate experimentally in six datasets and show that it improves the accuracy as well as the robustness of the models to adversarial perturbations.
  arXiv  Detail & Related papers  (2022-02-10T14:54:29Z)
• Stabilizing Equilibrium Models by Jacobian Regularization [151.78151873928027]
  Deep equilibrium networks (DEQs) are a new class of models that eschews traditional depth in favor of finding the fixed point of a single nonlinear layer. We propose a regularization scheme for DEQ models that explicitly regularizes the Jacobian of the fixed-point update equations to stabilize the learning of equilibrium models. We show that this regularization adds only minimal computational cost, significantly stabilizes the fixed-point convergence in both forward and backward passes, and scales well to high-dimensional, realistic domains.
  arXiv  Detail & Related papers  (2021-06-28T00:14:11Z)
• Robust Implicit Networks via Non-Euclidean Contractions [63.91638306025768]
  Implicit neural networks show improved accuracy and significant reduction in memory consumption. They can suffer from ill-posedness and convergence instability. This paper provides a new framework to design well-posed and robust implicit neural networks.
  arXiv  Detail & Related papers  (2021-06-06T18:05:02Z)
• Recurrent Equilibrium Networks: Flexible Dynamic Models with Guaranteed Stability and Robustness [3.2872586139884623]
  This paper introduces recurrent equilibrium networks (RENs) for applications in machine learning, system identification and control. RENs are parameterized directly by quadratic vector in RN, i.e. stability and robustness are ensured without parameter constraints. The paper also presents applications in data-driven nonlinear observer design and control with stability guarantees.
  arXiv  Detail & Related papers  (2021-04-13T05:09:41Z)
• Lipschitz Recurrent Neural Networks [100.72827570987992]
  We show that our Lipschitz recurrent unit is more robust with respect to input and parameter perturbations as compared to other continuous-time RNNs. Our experiments demonstrate that the Lipschitz RNN can outperform existing recurrent units on a range of benchmark tasks.
  arXiv  Detail & Related papers  (2020-06-22T08:44:52Z)
• Monotone operator equilibrium networks [97.86610752856987]
  We develop a new class of implicit-depth model based on the theory of monotone operators, the Monotone Operator Equilibrium Network (monDEQ) We show the close connection between finding the equilibrium point of an implicit network and solving a form of monotone operator splitting problem. We then develop a parameterization of the network which ensures that all operators remain monotone, which guarantees the existence of a unique equilibrium point.
  arXiv  Detail & Related papers  (2020-06-15T17:57:31Z)
• Mathematical foundations of stable RKHSs [1.52292571922932]
  Reproducing kernel Hilbert spaces (RKHSs) are key spaces for machine learning that are becoming popular also for linear system identification. In this paper we provide new structural properties of stable RKHSs.
  arXiv  Detail & Related papers  (2020-05-06T17:25:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.