Dimension of activity in random neural networks

• URL: http://arxiv.org/abs/2207.12373v3
• Date: Mon, 11 Sep 2023 18:15:09 GMT
• Title: Dimension of activity in random neural networks
• Authors: David G. Clark, L.F. Abbott, Ashok Litwin-Kumar
• Abstract summary: Neural networks are high-dimensional nonlinear dynamical systems that process information through the coordinated activity of many connected units. We calculate cross covariances self-consistently via a two-site cavity DMFT. Our formulae apply to a wide range of single-unit dynamics and generalize to non-i.i.d. couplings.
• Score: 6.752538702870792
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Neural networks are high-dimensional nonlinear dynamical systems that process information through the coordinated activity of many connected units. Understanding how biological and machine-learning networks function and learn requires knowledge of the structure of this coordinated activity, information contained, for example, in cross covariances between units. Self-consistent dynamical mean field theory (DMFT) has elucidated several features of random neural networks -- in particular, that they can generate chaotic activity -- however, a calculation of cross covariances using this approach has not been provided. Here, we calculate cross covariances self-consistently via a two-site cavity DMFT. We use this theory to probe spatiotemporal features of activity coordination in a classic random-network model with independent and identically distributed (i.i.d.) couplings, showing an extensive but fractionally low effective dimension of activity and a long population-level timescale. Our formulae apply to a wide range of single-unit dynamics and generalize to non-i.i.d. couplings. As an example of the latter, we analyze the case of partially symmetric couplings.

Related papers

• Connectivity structure and dynamics of nonlinear recurrent neural networks [46.62658917638706]
  We develop a theory to analyze how structure in connectivity shapes the high-dimensional, internally generated activity of neural networks. Our theory provides tools to relate neural-network architecture and collective dynamics in artificial and biological systems.
  arXiv  Detail & Related papers  (2024-09-03T15:08:37Z)
• Heterogenous Memory Augmented Neural Networks [84.29338268789684]
  We introduce a novel heterogeneous memory augmentation approach for neural networks. By introducing learnable memory tokens with attention mechanism, we can effectively boost performance without huge computational overhead. We show our approach on various image and graph-based tasks under both in-distribution (ID) and out-of-distribution (OOD) conditions.
  arXiv  Detail & Related papers  (2023-10-17T01:05:28Z)
• How neural networks learn to classify chaotic time series [77.34726150561087]
  We study the inner workings of neural networks trained to classify regular-versus-chaotic time series. We find that the relation between input periodicity and activation periodicity is key for the performance of LKCNN models.
  arXiv  Detail & Related papers  (2023-06-04T08:53:27Z)
• Sparse Interaction Additive Networks via Feature Interaction Detection and Sparse Selection [10.191597755296163]
  We develop a tractable selection algorithm to efficiently identify the necessary feature combinations. Our proposed Sparse Interaction Additive Networks (SIAN) construct a bridge from simple and interpretable models to fully connected neural networks.
  arXiv  Detail & Related papers  (2022-09-19T19:57:17Z)
• Imaginary components of out-of-time correlators and information scrambling for navigating the learning landscape of a quantum machine learning model [0.0]
  We analytically illustrate that hitherto unexplored imaginary components of out-of-time correlators can provide unprecedented insight into the information scrambling capacity of a graph neural network. Such an analysis demystifies the training of quantum machine learning models by unraveling how quantum information is scrambled through such a network.
  arXiv  Detail & Related papers  (2022-08-29T06:17:28Z)
• Bayesian Inference of Stochastic Dynamical Networks [0.0]
  This paper presents a novel method for learning network topology and internal dynamics. It is compared with group sparse Bayesian learning (GSBL), BINGO, kernel-based methods, dynGENIE3, GENIE3 and ARNI. Our method achieves state-of-the-art performance compared with group sparse Bayesian learning (GSBL), BINGO, kernel-based methods, dynGENIE3, GENIE3 and ARNI.
  arXiv  Detail & Related papers  (2022-06-02T03:22:34Z)
• Decomposing neural networks as mappings of correlation functions [57.52754806616669]
  We study the mapping between probability distributions implemented by a deep feed-forward network. We identify essential statistics in the data, as well as different information representations that can be used by neural networks.
  arXiv  Detail & Related papers  (2022-02-10T09:30:31Z)
• Learning Interpretable Models for Coupled Networks Under Domain Constraints [8.308385006727702]
  We investigate the idea of coupled networks by focusing on interactions between structural edges and functional edges of brain networks. We propose a novel formulation to place hard network constraints on the noise term while estimating interactions. We validate our method on multishell diffusion and task-evoked fMRI datasets from the Human Connectome Project.
  arXiv  Detail & Related papers  (2021-04-19T06:23:31Z)
• Towards Interaction Detection Using Topological Analysis on Neural Networks [55.74562391439507]
  In neural networks, any interacting features must follow a strongly weighted connection to common hidden units. We propose a new measure for quantifying interaction strength, based upon the well-received theory of persistent homology. A Persistence Interaction detection(PID) algorithm is developed to efficiently detect interactions.
  arXiv  Detail & Related papers  (2020-10-25T02:15:24Z)
• Provably Efficient Neural Estimation of Structural Equation Model: An Adversarial Approach [144.21892195917758]
  We study estimation in a class of generalized Structural equation models (SEMs) We formulate the linear operator equation as a min-max game, where both players are parameterized by neural networks (NNs), and learn the parameters of these neural networks using a gradient descent. For the first time we provide a tractable estimation procedure for SEMs based on NNs with provable convergence and without the need for sample splitting.
  arXiv  Detail & Related papers  (2020-07-02T17:55:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.