Lyapunov-Stable Deep Equilibrium Models

• URL: http://arxiv.org/abs/2304.12707v3
• Date: Wed, 10 Jan 2024 08:16:48 GMT
• Title: Lyapunov-Stable Deep Equilibrium Models
• Authors: Haoyu Chu, Shikui Wei, Ting Liu, Yao Zhao and Yuto Miyatake
• Abstract summary: We propose a robust DEQ model with guaranteed provable stability via Lyapunov theory. We evaluate LyaDEQ models under well-known adversarial attacks. We show that the LyaDEQ model can be combined with other defense methods, such as adversarial training, to achieve even better robustness.
• Score: 47.62037001903746
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Deep equilibrium (DEQ) models have emerged as a promising class of implicit layer models, which abandon traditional depth by solving for the fixed points of a single nonlinear layer. Despite their success, the stability of the fixed points for these models remains poorly understood. By considering DEQ models as nonlinear dynamic systems, we propose a robust DEQ model named LyaDEQ with guaranteed provable stability via Lyapunov theory. The crux of our method is ensuring the Lyapunov stability of the DEQ model's fixed points, which enables the proposed model to resist minor initial perturbations. To avoid poor adversarial defense due to Lyapunov-stable fixed points being located near each other, we orthogonalize the layers after the Lyapunov stability module to separate different fixed points. We evaluate LyaDEQ models under well-known adversarial attacks, and experimental results demonstrate significant improvement in robustness. Furthermore, we show that the LyaDEQ model can be combined with other defense methods, such as adversarial training, to achieve even better adversarial robustness.

Related papers

• Distributionally Robust Policy and Lyapunov-Certificate Learning [13.38077406934971]
  Key challenge in designing controllers with stability guarantees for uncertain systems is the accurate determination of and adaptation to shifts in model parametric uncertainty during online deployment. We tackle this with a novel distributionally robust formulation of the Lyapunov derivative chance constraint ensuring a monotonic decrease of the Lyapunov certificate. We show that, for the resulting closed-loop system, the global stability of its equilibrium can be certified with high confidence, even with Out-of-Distribution uncertainties.
  arXiv  Detail & Related papers  (2024-04-03T18:57:54Z)
• Extreme Miscalibration and the Illusion of Adversarial Robustness [66.29268991629085]
  Adversarial Training is often used to increase model robustness. We show that this observed gain in robustness is an illusion of robustness (IOR) We urge the NLP community to incorporate test-time temperature scaling into their robustness evaluations.
  arXiv  Detail & Related papers  (2024-02-27T13:49:12Z)
• Positive concave deep equilibrium models [7.148312060227714]
  Deep equilibrium (DEQ) models are a memory efficient alternative to standard neural networks. We introduce a novel class of DEQ models called positive concave deep equilibrium (pcDEQ) models. Our approach, which is based on nonlinear Perron-Frobenius theory, enforces nonnegative weights and activation functions that are concave on the positive orthant.
  arXiv  Detail & Related papers  (2024-02-06T14:24:29Z)
• The Risk of Federated Learning to Skew Fine-Tuning Features and Underperform Out-of-Distribution Robustness [50.52507648690234]
  Federated learning has the risk of skewing fine-tuning features and compromising the robustness of the model. We introduce three robustness indicators and conduct experiments across diverse robust datasets. Our approach markedly enhances the robustness across diverse scenarios, encompassing various parameter-efficient fine-tuning methods.
  arXiv  Detail & Related papers  (2024-01-25T09:18:51Z)
• Improving Adversarial Robustness of DEQs with Explicit Regulations Along the Neural Dynamics [26.94367957377311]
  Deep equilibrium (DEQ) models replace the multiple-layer stacking of conventional deep networks with a fixed-point iteration of a single-layer transformation. Existing works improve the robustness of general DEQ models with the widely-used adversarial training (AT) framework, but they fail to exploit the structural uniquenesses of DEQ models. We propose reducing prediction entropy by progressively updating inputs along the neural dynamics. Our methods substantially increase the robustness of DEQ models and even outperform the strong deep network baselines.
  arXiv  Detail & Related papers  (2023-06-02T10:49:35Z)
• KCRL: Krasovskii-Constrained Reinforcement Learning with Guaranteed Stability in Nonlinear Dynamical Systems [66.9461097311667]
  We propose a model-based reinforcement learning framework with formal stability guarantees. The proposed method learns the system dynamics up to a confidence interval using feature representation. We show that KCRL is guaranteed to learn a stabilizing policy in a finite number of interactions with the underlying unknown system.
  arXiv  Detail & Related papers  (2022-06-03T17:27:04Z)
• Stable Neural ODE with Lyapunov-Stable Equilibrium Points for Defending Against Adversarial Attacks [32.88499015927756]
  We propose a stable neural ODE with Lyapunov-stable equilibrium points for defending against adversarial attacks (SODEF) We provide theoretical results that give insights into the stability of SODEF as well as the choice of regularizers to ensure its stability.
  arXiv  Detail & Related papers  (2021-10-25T14:09:45Z)
• Stabilizing Equilibrium Models by Jacobian Regularization [151.78151873928027]
  Deep equilibrium networks (DEQs) are a new class of models that eschews traditional depth in favor of finding the fixed point of a single nonlinear layer. We propose a regularization scheme for DEQ models that explicitly regularizes the Jacobian of the fixed-point update equations to stabilize the learning of equilibrium models. We show that this regularization adds only minimal computational cost, significantly stabilizes the fixed-point convergence in both forward and backward passes, and scales well to high-dimensional, realistic domains.
  arXiv  Detail & Related papers  (2021-06-28T00:14:11Z)
• Almost Surely Stable Deep Dynamics [4.199844472131922]
  We introduce a method for learning provably stable deep neural network based dynamic models from observed data. Our method works by embedding a Lyapunov neural network into the dynamic model, thereby inherently satisfying the stability criterion.
  arXiv  Detail & Related papers  (2021-03-26T20:37:08Z)
• Efficient Empowerment Estimation for Unsupervised Stabilization [75.32013242448151]
  empowerment principle enables unsupervised stabilization of dynamical systems at upright positions. We propose an alternative solution based on a trainable representation of a dynamical system as a Gaussian channel. We show that our method has a lower sample complexity, is more stable in training, possesses the essential properties of the empowerment function, and allows estimation of empowerment from images.
  arXiv  Detail & Related papers  (2020-07-14T21:10:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.