Self-organized criticality in neural networks

• URL: http://arxiv.org/abs/2107.03402v1
• Date: Wed, 7 Jul 2021 18:00:03 GMT
• Title: Self-organized criticality in neural networks
• Authors: Mikhail I. Katsnelson, Vitaly Vanchurin, Tom Westerhout
• Abstract summary: We show that learning dynamics of neural networks is generically attracted towards a self-organized critical state. Our results support the claim that the universe might be a neural network.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We demonstrate, both analytically and numerically, that learning dynamics of neural networks is generically attracted towards a self-organized critical state. The effect can be modeled with quartic interactions between non-trainable variables (e.g. states of neurons) and trainable variables (e.g. weight matrix). Non-trainable variables are rapidly driven towards stochastic equilibrium and trainable variables are slowly driven towards learning equilibrium described by a scale-invariant distribution on a wide range of scales. Our results suggest that the scale invariance observed in many physical and biological systems might be due to some kind of learning dynamics and support the claim that the universe might be a neural network.

Related papers

• Expressivity of Neural Networks with Random Weights and Learned Biases [44.02417750529102]
  Recent work has pushed the bounds of universal approximation by showing that arbitrary functions can similarly be learned by tuning smaller subsets of parameters. We provide theoretical and numerical evidence demonstrating that feedforward neural networks with fixed random weights can be trained to perform multiple tasks by learning biases only. Our results are relevant to neuroscience, where they demonstrate the potential for behaviourally relevant changes in dynamics without modifying synaptic weights.
  arXiv  Detail & Related papers  (2024-07-01T04:25:49Z)
• Dataset-learning duality and emergent criticality [0.0]
  We show a duality map between a subspace of non-trainable variables and a subspace of trainable variables. We use the duality to study the emergence of criticality, or the power-law distributions of fluctuations of the trainable variables.
  arXiv  Detail & Related papers  (2024-05-27T17:44:33Z)
• Astrocytes as a mechanism for meta-plasticity and contextually-guided network function [2.66269503676104]
  Astrocytes are a ubiquitous and enigmatic type of non-neuronal cell. Astrocytes may play a more direct and active role in brain function and neural computation.
  arXiv  Detail & Related papers  (2023-11-06T20:31:01Z)
• Unsupervised Learning of Invariance Transformations [105.54048699217668]
  We develop an algorithmic framework for finding approximate graph automorphisms. We discuss how this framework can be used to find approximate automorphisms in weighted graphs in general.
  arXiv  Detail & Related papers  (2023-07-24T17:03:28Z)
• Machine learning in and out of equilibrium [58.88325379746631]
  Our study uses a Fokker-Planck approach, adapted from statistical physics, to explore these parallels. We focus in particular on the stationary state of the system in the long-time limit, which in conventional SGD is out of equilibrium. We propose a new variation of Langevin dynamics (SGLD) that harnesses without replacement minibatching.
  arXiv  Detail & Related papers  (2023-06-06T09:12:49Z)
• Equivalence of Additive and Multiplicative Coupling in Spiking Neural Networks [0.0]
  Spiking neural network models characterize the emergent collective dynamics of circuits of biological neurons. We show that spiking neural network models with additive coupling are equivalent to models with multiplicative coupling.
  arXiv  Detail & Related papers  (2023-03-31T20:19:11Z)
• Learning Physical Dynamics with Subequivariant Graph Neural Networks [99.41677381754678]
  Graph Neural Networks (GNNs) have become a prevailing tool for learning physical dynamics. Physical laws abide by symmetry, which is a vital inductive bias accounting for model generalization. Our model achieves on average over 3% enhancement in contact prediction accuracy across 8 scenarios on Physion and 2X lower rollout MSE on RigidFall.
  arXiv  Detail & Related papers  (2022-10-13T10:00:30Z)
• Towards a theory of quantum gravity from neural networks [0.0]
  We show that the non-equilibrium dynamics of trainable variables can be described by the Madelung equations. We argue that the Lorentz symmetries and curved space-time can emerge from the interplay between entropy production and entropy destruction due to learning.
  arXiv  Detail & Related papers  (2021-10-28T12:39:01Z)
• Dynamic Neural Diversification: Path to Computationally Sustainable Neural Networks [68.8204255655161]
  Small neural networks with a constrained number of trainable parameters, can be suitable resource-efficient candidates for many simple tasks. We explore the diversity of the neurons within the hidden layer during the learning process. We analyze how the diversity of the neurons affects predictions of the model.
  arXiv  Detail & Related papers  (2021-09-20T15:12:16Z)
• Artificial Neural Variability for Deep Learning: On Overfitting, Noise Memorization, and Catastrophic Forgetting [135.0863818867184]
  artificial neural variability (ANV) helps artificial neural networks learn some advantages from natural'' neural networks. ANV plays as an implicit regularizer of the mutual information between the training data and the learned model. It can effectively relieve overfitting, label noise memorization, and catastrophic forgetting at negligible costs.
  arXiv  Detail & Related papers  (2020-11-12T06:06:33Z)
• Spherical Motion Dynamics: Learning Dynamics of Neural Network with Normalization, Weight Decay, and SGD [105.99301967452334]
  We show the learning dynamics of neural network with normalization, weight decay (WD), and SGD (with momentum) named as Spherical Motion Dynamics (SMD) We verify our assumptions and theoretical results on various computer vision tasks including ImageNet and MSCOCO with standard settings.
  arXiv  Detail & Related papers  (2020-06-15T14:16:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.