Dynamics and Representation Structure of Local Approximations to Gradient-Based Learning in Linear Recurrent Neural Networks

• URL: http://arxiv.org/abs/2606.00243v1
• Date: Fri, 29 May 2026 18:19:45 GMT
• Title: Dynamics and Representation Structure of Local Approximations to Gradient-Based Learning in Linear Recurrent Neural Networks
• Abstract summary: We study the learning dynamics of random feedback local online (RFLO) and truncated backpropagation through time (tBPTT)<n>Our work provides analytical insight into how locality constraints shape learning dynamics, with implications for neuroscientific models of learning and alternative optimization approaches for RNNs.
• Score: 56.42974363744522
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Biological and neuromorphic recurrent neural networks (RNNs) are subject to spatial and temporal locality constraints on the information that can plausibly be used during learning. A common strategy to satisfy these constraints is to modify gradient descent by neglecting non-local terms to varying degrees, as in random feedback local online (RFLO) learning and truncated backpropagation through time (tBPTT). However, the learning dynamics of these algorithms, and how they compare with BPTT, remain poorly understood. We apply dynamical systems theory to data-aligned linear RNNs -- whose dynamics can be separated into orthogonal modes -- to compare stationary solutions, stability properties, and convergence rates, finding qualitatively distinct behaviour for RFLO versus BPTT and one-step tBPTT. We further observe that the solutions learned by RFLO are restricted to low-rank perturbations of initial parameters, a result which holds beyond the data-aligned setting. Our work provides analytical insight into how locality constraints shape learning dynamics, with implications for neuroscientific models of learning and alternative optimization approaches for RNNs.

Related papers

• When Learning Hurts: Fixed-Pole RNN for Real-Time Online Training [58.25341036646294]
  We analytically examine why learning recurrent poles does not provide tangible benefits in data and empirically offer real-time learning scenarios.<n>We show that fixed-pole networks achieve superior performance with lower training complexity, making them more suitable for online real-time tasks.
  arXiv  Detail & Related papers  (2026-02-25T00:15:13Z)
• A Low-complexity Structured Neural Network to Realize States of Dynamical Systems [0.0]
  This paper stems from data-driven learning to advance states of dynamical systems utilizing a structured neural network (StNN)<n>We present numerical simulations to solve dynamical systems utilizing the StNN based on the Hankel operator.<n>We show that the proposed StNN paves way for realizing state-space dynamical systems with a low-complexity learning enabling prediction and understanding of future states.
  arXiv  Detail & Related papers  (2025-03-31T03:52:38Z)
• TESS: A Scalable Temporally and Spatially Local Learning Rule for Spiking Neural Networks [6.805933498669221]
  Training neural networks (SNNs) on resource-constrained devices remains challenging due to high computational and memory demands.<n>We introduce TESS, a temporally and spatially local learning rule for training SNNs.<n>Our approach addresses both temporal and spatial credit assignments by relying solely on locally available signals within each neuron.
  arXiv  Detail & Related papers  (2025-02-03T21:23:15Z)
• Towards Model Discovery Using Domain Decomposition and PINNs [44.99833362998488]
  The study evaluates the performance of two approaches, namely Physics-Informed Neural Networks (PINNs) and Finite Basis Physics-Informed Neural Networks (FBPINNs) We find a better performance for the FBPINN approach compared to the vanilla PINN approach, even in cases with data from only a quasi-stationary time domain with few dynamics.
  arXiv  Detail & Related papers  (2024-10-02T14:38:37Z)
• Analysing Rescaling, Discretization, and Linearization in RNNs for Neural System Modelling [0.0]
  Recurrent Neural Networks (RNNs) are widely used for modelling neural activity, yet the mathematical interplay of core procedures is uncharacterized.<n>This study establishes the conditions under which these procedures commute, enabling flexible application in computational neuroscience.<n>Our findings directly guide the design of biologically plausible RNNs for simulating neural dynamics in decision-making and motor control.
  arXiv  Detail & Related papers  (2023-12-26T10:00:33Z)
• Provably Efficient Neural Estimation of Structural Equation Model: An Adversarial Approach [144.21892195917758]
  We study estimation in a class of generalized Structural equation models (SEMs) We formulate the linear operator equation as a min-max game, where both players are parameterized by neural networks (NNs), and learn the parameters of these neural networks using a gradient descent. For the first time we provide a tractable estimation procedure for SEMs based on NNs with provable convergence and without the need for sample splitting.
  arXiv  Detail & Related papers  (2020-07-02T17:55:47Z)
• An Ode to an ODE [78.97367880223254]
  We present a new paradigm for Neural ODE algorithms, called ODEtoODE, where time-dependent parameters of the main flow evolve according to a matrix flow on the group O(d) This nested system of two flows provides stability and effectiveness of training and provably solves the gradient vanishing-explosion problem.
  arXiv  Detail & Related papers  (2020-06-19T22:05:19Z)
• Network Diffusions via Neural Mean-Field Dynamics [52.091487866968286]
  We propose a novel learning framework for inference and estimation problems of diffusion on networks. Our framework is derived from the Mori-Zwanzig formalism to obtain an exact evolution of the node infection probabilities. Our approach is versatile and robust to variations of the underlying diffusion network models.
  arXiv  Detail & Related papers  (2020-06-16T18:45:20Z)
• Rectified Linear Postsynaptic Potential Function for Backpropagation in Deep Spiking Neural Networks [55.0627904986664]
  Spiking Neural Networks (SNNs) usetemporal spike patterns to represent and transmit information, which is not only biologically realistic but also suitable for ultra-low-power event-driven neuromorphic implementation. This paper investigates the contribution of spike timing dynamics to information encoding, synaptic plasticity and decision making, providing a new perspective to design of future DeepSNNs and neuromorphic hardware systems.
  arXiv  Detail & Related papers  (2020-03-26T11:13:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.