Random matrix theory of sparse neuronal networks with heterogeneous timescales

• URL: http://arxiv.org/abs/2512.12767v1
• Date: Sun, 14 Dec 2025 17:02:22 GMT
• Title: Random matrix theory of sparse neuronal networks with heterogeneous timescales
• Authors: Thiparat Chotibut, Oleg Evnin, Weerawit Horinouchi,
• Abstract summary: Training recurrent neuronal networks consists of excitatory (E) and inhibitory (I) units with additive noise for working memory computation.<n>Here, we investigate the dynamics near these equilibria and show that they are sparse, non-Hermitian rectangular-block matrices modified by heterogeneous synaptic decay timescales and activation-function gains.<n>An analytic description of the spectral edge is obtained, relating statistical parameters of the Jacobians to near-critical features of the equilibria essential for robust working memory computation.
• Score: 0.6181093777643575
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Training recurrent neuronal networks consisting of excitatory (E) and inhibitory (I) units with additive noise for working memory computation slows and diversifies inhibitory timescales, leading to improved task performance that is attributed to emergent marginally stable equilibria [PNAS 122 (2025) e2316745122]. Yet the link between trained network characteristics and their roles in shaping desirable dynamical landscapes remains unexplored. Here, we investigate the Jacobian matrices describing the dynamics near these equilibria and show that they are sparse, non-Hermitian rectangular-block matrices modified by heterogeneous synaptic decay timescales and activation-function gains. We specify a random matrix ensemble that faithfully captures the spectra of trained Jacobian matrices, arising from the inhibitory core - excitatory periphery network motif (pruned E weights, broadly distributed I weights) observed post-training. An analytic theory of this ensemble is developed using statistical field theory methods: a Hermitized resolvent representation of the spectral density processed with a supersymmetry-based treatment in the style of Fyodorov and Mirlin. In this manner, an analytic description of the spectral edge is obtained, relating statistical parameters of the Jacobians (sparsity, weight variances, E/I ratio, and the distributions of timescales and gains) to near-critical features of the equilibria essential for robust working memory computation.

Related papers

• Dispelling the Curse of Singularities in Neural Network Optimizations [22.05217959662069]
  We show that the gradient Frobenius norms are bounded by the top singular values of the weight matrices, and as training progresses, the mutually reinforcing growth of weight and representation singularities, relaxes these bounds, escalating the risk of sharp loss explosions.<n>To counter this, we propose Parametric Singularity Smoothing (PSS), a lightweight, flexible, effective method for smoothing the singular spectra of weight matrices.
  arXiv  Detail & Related papers  (2026-02-01T16:09:06Z)
• SIGMA: Scalable Spectral Insights for LLM Collapse [51.863164847253366]
  We introduce SIGMA (Spectral Inequalities for Gram Matrix Analysis), a unified framework for model collapse.<n>By utilizing benchmarks that deriving and deterministic bounds on the matrix's spectrum, SIGMA provides a mathematically grounded metric to track the contraction of the representation space.<n>We demonstrate that SIGMA effectively captures the transition towards states, offering both theoretical insights into the mechanics of collapse.
  arXiv  Detail & Related papers  (2026-01-06T19:47:11Z)
• Disordered Dynamics in High Dimensions: Connections to Random Matrices and Machine Learning [52.26396748560348]
  We provide an overview of high dimensional dynamical systems driven by random matrices.<n>We focus on applications to simple models of learning and generalization in machine learning theory.
  arXiv  Detail & Related papers  (2026-01-03T00:12:32Z)
• Learning Dynamics in Memristor-Based Equilibrium Propagation [0.7266320276728724]
  We investigate the effect of nonlinear, memristor-driven weight updates on the convergence behaviour of neural networks trained with equilibrium propagation (EqProp)<n>EqProp can achieve robust convergence under nonlinear weight updates, provided that memristors exhibit a sufficiently wide resistance range of at least an order of magnitude.
  arXiv  Detail & Related papers  (2025-12-13T18:57:05Z)
• Generative System Dynamics in Recurrent Neural Networks [56.958984970518564]
  We investigate the continuous time dynamics of Recurrent Neural Networks (RNNs)<n>We show that skew-symmetric weight matrices are fundamental to enable stable limit cycles in both linear and nonlinear configurations.<n> Numerical simulations showcase how nonlinear activation functions not only maintain limit cycles, but also enhance the numerical stability of the system integration process.
  arXiv  Detail & Related papers  (2025-04-16T10:39:43Z)
• Semi-group influence matrices for non-equilibrium quantum impurity models [0.0]
  We introduce a framework for describing the real-time dynamics of quantum impurity models out of equilibrium.<n>For a quantum impurity model with on-site two-fermion loss, we compute the spectral function and confirm the emergence of Kondo physics at large loss rates.
  arXiv  Detail & Related papers  (2025-01-31T19:00:42Z)
• Machine learning in and out of equilibrium [58.88325379746631]
  Our study uses a Fokker-Planck approach, adapted from statistical physics, to explore these parallels. We focus in particular on the stationary state of the system in the long-time limit, which in conventional SGD is out of equilibrium. We propose a new variation of Langevin dynamics (SGLD) that harnesses without replacement minibatching.
  arXiv  Detail & Related papers  (2023-06-06T09:12:49Z)
• Capturing dynamical correlations using implicit neural representations [85.66456606776552]
  We develop an artificial intelligence framework which combines a neural network trained to mimic simulated data from a model Hamiltonian with automatic differentiation to recover unknown parameters from experimental data. In doing so, we illustrate the ability to build and train a differentiable model only once, which then can be applied in real-time to multi-dimensional scattering data.
  arXiv  Detail & Related papers  (2023-04-08T07:55:36Z)
• Spectrum of non-Hermitian deep-Hebbian neural networks [3.333967282951668]
  We integrate the experimental observation of wide synaptic integration window into our model of sequence retrieval in the continuous time dynamics. Our work provides a systematic study of time-lagged correlations with arbitrary time delays, and thus can inspire future studies of a broad class of memory models.
  arXiv  Detail & Related papers  (2022-08-24T10:09:47Z)
• Out-of-time-order correlations and the fine structure of eigenstate thermalisation [58.720142291102135]
  Out-of-time-orderors (OTOCs) have become established as a tool to characterise quantum information dynamics and thermalisation. We show explicitly that the OTOC is indeed a precise tool to explore the fine details of the Eigenstate Thermalisation Hypothesis (ETH) We provide an estimation of the finite-size scaling of $omega_textrmGOE$ for the general class of observables composed of sums of local operators in the infinite-temperature regime.
  arXiv  Detail & Related papers  (2021-03-01T17:51:46Z)
• Modification of quantum many-body relaxation by perturbations exhibiting a banded matrix structure [0.0]
  We investigate how the observable relaxation behavior of an isolated quantum many-body system is modified in response to weak-to-moderate perturbations. A key role is played by the so-called perturbation profile, which characterizes the dependence of the perturbation matrix elements in the eigenbasis of the unperturbed Hamiltonian.
  arXiv  Detail & Related papers  (2020-08-09T15:29:01Z)
• Multiplicative noise and heavy tails in stochastic optimization [62.993432503309485]
  empirical optimization is central to modern machine learning, but its role in its success is still unclear. We show that it commonly arises in parameters of discrete multiplicative noise due to variance. A detailed analysis is conducted in which we describe on key factors, including recent step size, and data, all exhibit similar results on state-of-the-art neural network models.
  arXiv  Detail & Related papers  (2020-06-11T09:58:01Z)
• On dissipative symplectic integration with applications to gradient-based optimization [77.34726150561087]
  We propose a geometric framework in which discretizations can be realized systematically. We show that a generalization of symplectic to nonconservative and in particular dissipative Hamiltonian systems is able to preserve rates of convergence up to a controlled error.
  arXiv  Detail & Related papers  (2020-04-15T00:36:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.