Divisive Normalization Shapes Low-Rank Slow Manifolds for Continuous Working Memory

• URL: http://arxiv.org/abs/2608.01947v1
• Date: Mon, 03 Aug 2026 09:20:05 GMT
• Title: Divisive Normalization Shapes Low-Rank Slow Manifolds for Continuous Working Memory
• Abstract summary: We show that divisive normalization is mathematically essential to prevent manifold shattering under time-varying inputs.<n>Our findings identify divisive normalization as a critical computational mechanism for learning high-fidelity continuous representations.
• Score: 6.1311571881384745
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The ability to robustly maintain and update continuous variables is a hallmark of working memory. While classical continuous attractor networks suffer from severe fine-tuning fragility, standard artificial recurrent neural networks (RNNs) like GRUs and LSTMs typically fail to stably learn continuous manifolds, instead shattering the state space into discretized point attractors. To bridge this gap, we draw inspiration from divisive normalization, a canonical neural computation widely observed across cortical circuits, and propose the Recurrent Divisive Normalization Network (RDNN), a minimal and algebraically isolated model of dynamic division. Through dynamical systems analysis on canonical working memory tasks, we demonstrate that this biophysical constraint allows the network to converge to robust, high-fidelity slow manifolds. Furthermore, we analyze the gradient dynamics of divisive normalization during Backpropagation Through Time (BPTT), showing that it introduces an activity-dependent local gradient scaling. This scaling dampens parameter updates in highly active regimes, which empirically aligns with a significant self-compression of the network's effective rank, confining the recurrent dynamics to a tight, low-dimensional subspace while avoiding the optimization pathologies associated with explicit low-rank factorization. Finally, ablations demonstrate that while subtractive inhibition can maintain static memories, divisive normalization is mathematically essential to prevent manifold shattering under time-varying inputs. Our findings identify divisive normalization not merely as a biological artifact, but as a critical computational mechanism for learning high-fidelity continuous representations.

Related papers

• Multi-Timescale Conductance Spiking Networks: A Sparse, Gradient-Trainable Framework with Rich Firing Dynamics for Enhanced Temporal Processing [0.20240089219927163]
  Spiking neural networks (SNNs) promise low-power event-driven computation for temporally rich tasks.<n>We introduce multi-timescale conductance spiking networks, a gradient-trainable framework in which neural dynamics emerge from shaping the current-voltage curve.<n>We derive a discrete-time formulation of these differentiable dynamics, enabling direct backpropagation through time without surrogate-gradient approximations.
  arXiv  Detail & Related papers  (2026-05-12T09:22:45Z)
• Local Truncation Error-Guided Neural ODEs for Large Scale Traffic Forecasting [26.116856080963966]
  Local Truncation Error-Guided Neural ODEs (LTE-ODE)<n>Our architecture gracefully preserves high-precision continuous ODE evolution in stable regions.<n>LTE-ODE achieves state-of-the-art performance on multiple large-scale benchmarks.
  arXiv  Detail & Related papers  (2026-05-05T05:46:00Z)
• Unleashing Temporal Capacity of Spiking Neural Networks through Spatiotemporal Separation [67.69345363409835]
  Spiking Neural Networks (SNNs) are considered naturally suited for temporal processing, with membrane potential propagation widely regarded as the core temporal modeling mechanism.<n>We design Non-Stateful (NS) models progressively removing membrane propagation to its stage-wise role. Experiments reveal a counterintuitive phenomenon: moderate removal in shallow layers improves performance, while excessive removal causes collapse.
  arXiv  Detail & Related papers  (2025-12-05T07:05:53Z)
• Learning by Steering the Neural Dynamics: A Statistical Mechanics Perspective [0.0]
  We study how neural dynamics can support fully local, distributed learning.<n>We propose a biologically plausible algorithm for supervised learning with any binary recurrent network.
  arXiv  Detail & Related papers  (2025-10-13T22:28:34Z)
• Fractional Spike Differential Equations Neural Network with Efficient Adjoint Parameters Training [63.3991315762955]
  Spiking Neural Networks (SNNs) draw inspiration from biological neurons to create realistic models for brain-like computation.<n>Most existing SNNs assume a single time constant for neuronal membrane voltage dynamics, modeled by first-order ordinary differential equations (ODEs) with Markovian characteristics.<n>We propose the Fractional SPIKE Differential Equation neural network (fspikeDE), which captures long-term dependencies in membrane voltage and spike trains through fractional-order dynamics.
  arXiv  Detail & Related papers  (2025-07-22T18:20:56Z)
• Generative System Dynamics in Recurrent Neural Networks [56.958984970518564]
  We investigate the continuous time dynamics of Recurrent Neural Networks (RNNs)<n>We show that skew-symmetric weight matrices are fundamental to enable stable limit cycles in both linear and nonlinear configurations.<n> Numerical simulations showcase how nonlinear activation functions not only maintain limit cycles, but also enhance the numerical stability of the system integration process.
  arXiv  Detail & Related papers  (2025-04-16T10:39:43Z)
• Temporal Reversal Regularization for Spiking Neural Networks: Hybrid Spatio-Temporal Invariance for Generalization [3.7748662901422807]
  Spiking neural networks (SNNs) have received widespread attention as an ultra-low power computing paradigm.<n>Recent studies have shown that SNNs suffer from severe overfitting, which limits their generalization performance.<n>We propose a simple yet effective Temporal Reversal Regularization to mitigate overfitting during training and facilitate generalization of SNNs.
  arXiv  Detail & Related papers  (2024-08-17T06:23:38Z)
• Detecting Anomalies in Dynamic Graphs via Memory enhanced Normality [39.476378833827184]
  Anomaly detection in dynamic graphs presents a significant challenge due to the temporal evolution of graph structures and attributes. We introduce a novel spatial- temporal memories-enhanced graph autoencoder (STRIPE) STRIPE significantly outperforms existing methods with 5.8% improvement in AUC scores and 4.62X faster in training time.
  arXiv  Detail & Related papers  (2024-03-14T02:26:10Z)
• Reducing Catastrophic Forgetting in Self Organizing Maps with Internally-Induced Generative Replay [67.50637511633212]
  A lifelong learning agent is able to continually learn from potentially infinite streams of pattern sensory data. One major historic difficulty in building agents that adapt is that neural systems struggle to retain previously-acquired knowledge when learning from new samples. This problem is known as catastrophic forgetting (interference) and remains an unsolved problem in the domain of machine learning to this day.
  arXiv  Detail & Related papers  (2021-12-09T07:11:14Z)
• Limited-angle tomographic reconstruction of dense layered objects by dynamical machine learning [68.9515120904028]
  Limited-angle tomography of strongly scattering quasi-transparent objects is a challenging, highly ill-posed problem. Regularizing priors are necessary to reduce artifacts by improving the condition of such problems. We devised a recurrent neural network (RNN) architecture with a novel split-convolutional gated recurrent unit (SC-GRU) as the building block.
  arXiv  Detail & Related papers  (2020-07-21T11:48:22Z)
• Untangling tradeoffs between recurrence and self-attention in neural networks [81.30894993852813]
  We present a formal analysis of how self-attention affects gradient propagation in recurrent networks. We prove that it mitigates the problem of vanishing gradients when trying to capture long-term dependencies. We propose a relevancy screening mechanism that allows for a scalable use of sparse self-attention with recurrence.
  arXiv  Detail & Related papers  (2020-06-16T19:24:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.