Optimal input representation in neural systems at the edge of chaos
- URL: http://arxiv.org/abs/2107.05709v1
- Date: Mon, 12 Jul 2021 19:55:03 GMT
- Title: Optimal input representation in neural systems at the edge of chaos
- Authors: Guillermo B. Morales and Miguel A. Mu\~noz
- Abstract summary: We build an artificial neural network and train it to classify images.
We find that the best performance in such a task is obtained when the network operates near the critical point.
We conclude that operating near criticality can have -- besides the usually alleged virtues -- the advantage of allowing for flexible, robust and efficient input representations.
- Score: 0.0
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: Shedding light onto how biological systems represent, process and store
information in noisy environments is a key and challenging goal. A stimulating,
though controversial, hypothesis poses that operating in dynamical regimes near
the edge of a phase transition, i.e. at criticality or the "edge of chaos", can
provide information-processing living systems with important operational
advantages, creating, e.g., an optimal trade-off between robustness and
flexibility. Here, we elaborate on a recent theoretical result, which
establishes that the spectrum of covariance matrices of neural networks
representing complex inputs in a robust way needs to decay as a power-law of
the rank, with an exponent close to unity, a result that has been indeed
experimentally verified in neurons of the mouse visual cortex. Aimed at
understanding and mimicking these results, we construct an artificial neural
network and train it to classify images. Remarkably, we find that the best
performance in such a task is obtained when the network operates near the
critical point, at which the eigenspectrum of the covariance matrix follows the
very same statistics as actual neurons do. Thus, we conclude that operating
near criticality can also have -- besides the usually alleged virtues -- the
advantage of allowing for flexible, robust and efficient input representations.
Related papers
- Coding schemes in neural networks learning classification tasks [52.22978725954347]
We investigate fully-connected, wide neural networks learning classification tasks.
We show that the networks acquire strong, data-dependent features.
Surprisingly, the nature of the internal representations depends crucially on the neuronal nonlinearity.
arXiv Detail & Related papers (2024-06-24T14:50:05Z) - Exploring mechanisms of Neural Robustness: probing the bridge between geometry and spectrum [0.0]
We study the link between representation smoothness and spectrum by using weight, Jacobian and spectral regularization.
Our research aims to understand the interplay between geometry, spectral properties, robustness, and expressivity in neural representations.
arXiv Detail & Related papers (2024-02-05T12:06:00Z) - DISCOVER: Making Vision Networks Interpretable via Competition and
Dissection [11.028520416752325]
This work contributes to post-hoc interpretability, and specifically Network Dissection.
Our goal is to present a framework that makes it easier to discover the individual functionality of each neuron in a network trained on a vision task.
arXiv Detail & Related papers (2023-10-07T21:57:23Z) - Learning invariant representations of time-homogeneous stochastic dynamical systems [27.127773672738535]
We study the problem of learning a representation of the state that faithfully captures its dynamics.
This is instrumental to learning the transfer operator or the generator of the system.
We show that the search for a good representation can be cast as an optimization problem over neural networks.
arXiv Detail & Related papers (2023-07-19T11:32:24Z) - Inducing Gaussian Process Networks [80.40892394020797]
We propose inducing Gaussian process networks (IGN), a simple framework for simultaneously learning the feature space as well as the inducing points.
The inducing points, in particular, are learned directly in the feature space, enabling a seamless representation of complex structured domains.
We report on experimental results for real-world data sets showing that IGNs provide significant advances over state-of-the-art methods.
arXiv Detail & Related papers (2022-04-21T05:27:09Z) - Data-driven emergence of convolutional structure in neural networks [83.4920717252233]
We show how fully-connected neural networks solving a discrimination task can learn a convolutional structure directly from their inputs.
By carefully designing data models, we show that the emergence of this pattern is triggered by the non-Gaussian, higher-order local structure of the inputs.
arXiv Detail & Related papers (2022-02-01T17:11:13Z) - Latent Equilibrium: A unified learning theory for arbitrarily fast
computation with arbitrarily slow neurons [0.7340017786387767]
We introduce Latent Equilibrium, a new framework for inference and learning in networks of slow components.
We derive disentangled neuron and synapse dynamics from a prospective energy function.
We show how our principle can be applied to detailed models of cortical microcircuitry.
arXiv Detail & Related papers (2021-10-27T16:15:55Z) - Mitigating Performance Saturation in Neural Marked Point Processes:
Architectures and Loss Functions [50.674773358075015]
We propose a simple graph-based network structure called GCHP, which utilizes only graph convolutional layers.
We show that GCHP can significantly reduce training time and the likelihood ratio loss with interarrival time probability assumptions can greatly improve the model performance.
arXiv Detail & Related papers (2021-07-07T16:59:14Z) - Learning Connectivity of Neural Networks from a Topological Perspective [80.35103711638548]
We propose a topological perspective to represent a network into a complete graph for analysis.
By assigning learnable parameters to the edges which reflect the magnitude of connections, the learning process can be performed in a differentiable manner.
This learning process is compatible with existing networks and owns adaptability to larger search spaces and different tasks.
arXiv Detail & Related papers (2020-08-19T04:53:31Z) - Network Diffusions via Neural Mean-Field Dynamics [52.091487866968286]
We propose a novel learning framework for inference and estimation problems of diffusion on networks.
Our framework is derived from the Mori-Zwanzig formalism to obtain an exact evolution of the node infection probabilities.
Our approach is versatile and robust to variations of the underlying diffusion network models.
arXiv Detail & Related papers (2020-06-16T18:45:20Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.