Generative System Dynamics in Recurrent Neural Networks

• URL: http://arxiv.org/abs/2504.13951v1
• Date: Wed, 16 Apr 2025 10:39:43 GMT
• Title: Generative System Dynamics in Recurrent Neural Networks
• Authors: Michele Casoni, Tommaso Guidi, Alessandro Betti, Stefano Melacci, Marco Gori,
• Abstract summary: We investigate the continuous time dynamics of Recurrent Neural Networks (RNNs)<n>We show that skew-symmetric weight matrices are fundamental to enable stable limit cycles in both linear and nonlinear configurations.<n> Numerical simulations showcase how nonlinear activation functions not only maintain limit cycles, but also enhance the numerical stability of the system integration process.
• Score: 56.958984970518564
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this study, we investigate the continuous time dynamics of Recurrent Neural Networks (RNNs), focusing on systems with nonlinear activation functions. The objective of this work is to identify conditions under which RNNs exhibit perpetual oscillatory behavior, without converging to static fixed points. We establish that skew-symmetric weight matrices are fundamental to enable stable limit cycles in both linear and nonlinear configurations. We further demonstrate that hyperbolic tangent-like activation functions (odd, bounded, and continuous) preserve these oscillatory dynamics by ensuring motion invariants in state space. Numerical simulations showcase how nonlinear activation functions not only maintain limit cycles, but also enhance the numerical stability of the system integration process, mitigating those instabilities that are commonly associated with the forward Euler method. The experimental results of this analysis highlight practical considerations for designing neural architectures capable of capturing complex temporal dependencies, i.e., strategies for enhancing memorization skills in recurrent models.

Related papers

• Neural Contraction Metrics with Formal Guarantees for Discrete-Time Nonlinear Dynamical Systems [17.905596843865705]
  Contraction metrics provide a powerful framework for analyzing stability, robustness, and convergence of various dynamical systems. However, identifying these metrics for complex nonlinear systems remains an open challenge due to the lack of effective tools. This paper develops verifiable contraction metrics for discrete scalable nonlinear systems.
  arXiv  Detail & Related papers  (2025-04-23T21:27:32Z)
• Nonlinear Neural Dynamics and Classification Accuracy in Reservoir Computing [3.196204482566275]
  We study the accuracy of a reservoir computer in artificial classification tasks of varying complexity. We find that, even for activation functions with extremely reduced nonlinearity, weak recurrent interactions and small input signals, the reservoir is able to compute useful representations.
  arXiv  Detail & Related papers  (2024-11-15T08:52:12Z)
• Unconditional stability of a recurrent neural circuit implementing divisive normalization [0.0]
  We prove the remarkable property of unconditional local stability for an arbitrary-dimensional ORGaNICs circuit.<n>We show that ORGaNICs can be trained by backpropagation through time without gradient clipping/scaling.
  arXiv  Detail & Related papers  (2024-09-27T17:46:05Z)
• Analysing Rescaling, Discretization, and Linearization in RNNs for Neural System Modelling [0.0]
  Recurrent Neural Networks (RNNs) are widely used for modelling neural activity, yet the mathematical interplay of core procedures is uncharacterized.<n>This study establishes the conditions under which these procedures commute, enabling flexible application in computational neuroscience.<n>Our findings directly guide the design of biologically plausible RNNs for simulating neural dynamics in decision-making and motor control.
  arXiv  Detail & Related papers  (2023-12-26T10:00:33Z)
• Neural Abstractions [72.42530499990028]
  We present a novel method for the safety verification of nonlinear dynamical models that uses neural networks to represent abstractions of their dynamics. We demonstrate that our approach performs comparably to the mature tool Flow* on existing benchmark nonlinear models.
  arXiv  Detail & Related papers  (2023-01-27T12:38:09Z)
• Accelerated Continuous-Time Approximate Dynamic Programming via Data-Assisted Hybrid Control [0.0]
  We introduce an algorithm that incorporates dynamic momentum in actor-critic structures to control continuous-time dynamic plants with an affine structure in the input. By incorporating dynamic momentum in our algorithm, we are able to accelerate the convergence properties of the closed-loop system.
  arXiv  Detail & Related papers  (2022-04-27T05:36:51Z)
• Value Iteration in Continuous Actions, States and Time [99.00362538261972]
  We propose a continuous fitted value iteration (cFVI) algorithm for continuous states and actions. The optimal policy can be derived for non-linear control-affine dynamics. Videos of the physical system are available at urlhttps://sites.google.com/view/value-iteration.
  arXiv  Detail & Related papers  (2021-05-10T21:40:56Z)
• Lipschitz Recurrent Neural Networks [100.72827570987992]
  We show that our Lipschitz recurrent unit is more robust with respect to input and parameter perturbations as compared to other continuous-time RNNs. Our experiments demonstrate that the Lipschitz RNN can outperform existing recurrent units on a range of benchmark tasks.
  arXiv  Detail & Related papers  (2020-06-22T08:44:52Z)
• Liquid Time-constant Networks [117.57116214802504]
  We introduce a new class of time-continuous recurrent neural network models. Instead of declaring a learning system's dynamics by implicit nonlinearities, we construct networks of linear first-order dynamical systems. These neural networks exhibit stable and bounded behavior, yield superior expressivity within the family of neural ordinary differential equations.
  arXiv  Detail & Related papers  (2020-06-08T09:53:35Z)
• On dissipative symplectic integration with applications to gradient-based optimization [77.34726150561087]
  We propose a geometric framework in which discretizations can be realized systematically. We show that a generalization of symplectic to nonconservative and in particular dissipative Hamiltonian systems is able to preserve rates of convergence up to a controlled error.
  arXiv  Detail & Related papers  (2020-04-15T00:36:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.