Synergistic information supports modality integration and flexible
learning in neural networks solving multiple tasks
- URL: http://arxiv.org/abs/2210.02996v1
- Date: Thu, 6 Oct 2022 15:36:27 GMT
- Title: Synergistic information supports modality integration and flexible
learning in neural networks solving multiple tasks
- Authors: Alexandra M. Proca, Fernando E. Rosas, Andrea I. Luppi, Daniel Bor,
Matthew Crosby, Pedro A.M. Mediano
- Abstract summary: We investigate the information processing strategies adopted by simple artificial neural networks performing a variety of cognitive tasks.
Results show that synergy increases as neural networks learn multiple diverse tasks.
randomly turning off neurons during training through dropout increases network redundancy, corresponding to an increase in robustness.
- Score: 107.8565143456161
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Striking progress has recently been made in understanding human cognition by
analyzing how its neuronal underpinnings are engaged in different modes of
information processing. Specifically, neural information can be decomposed into
synergistic, redundant, and unique features, with synergistic components being
particularly aligned with complex cognition. However, two fundamental questions
remain unanswered: (a) precisely how and why a cognitive system can become
highly synergistic; and (b) how these informational states map onto artificial
neural networks in various learning modes. To address these questions, here we
employ an information-decomposition framework to investigate the information
processing strategies adopted by simple artificial neural networks performing a
variety of cognitive tasks in both supervised and reinforcement learning
settings. Our results show that synergy increases as neural networks learn
multiple diverse tasks. Furthermore, performance in tasks requiring integration
of multiple information sources critically relies on synergistic neurons.
Finally, randomly turning off neurons during training through dropout increases
network redundancy, corresponding to an increase in robustness. Overall, our
results suggest that while redundant information is required for robustness to
perturbations in the learning process, synergistic information is used to
combine information from multiple modalities -- and more generally for flexible
and efficient learning. These findings open the door to new ways of
investigating how and why learning systems employ specific
information-processing strategies, and support the principle that the capacity
for general-purpose learning critically relies in the system's information
dynamics.
Related papers
- From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networks [47.13391046553908]
In artificial networks, the effectiveness of these models relies on their ability to build task specific representation.
Prior studies highlight that different initializations can place networks in either a lazy regime, where representations remain static, or a rich/feature learning regime, where representations evolve dynamically.
These solutions capture the evolution of representations and the Neural Kernel across the spectrum from the rich to the lazy regimes.
arXiv Detail & Related papers (2024-09-22T23:19:04Z) - Enhancing learning in spiking neural networks through neuronal heterogeneity and neuromodulatory signaling [52.06722364186432]
We propose a biologically-informed framework for enhancing artificial neural networks (ANNs)
Our proposed dual-framework approach highlights the potential of spiking neural networks (SNNs) for emulating diverse spiking behaviors.
We outline how the proposed approach integrates brain-inspired compartmental models and task-driven SNNs, bioinspiration and complexity.
arXiv Detail & Related papers (2024-07-05T14:11:28Z) - Advanced Computing and Related Applications Leveraging Brain-inspired
Spiking Neural Networks [0.0]
Spiking neural network is one of the cores of artificial intelligence which realizes brain-like computing.
This paper summarizes the strengths, weaknesses and applicability of five neuronal models and analyzes the characteristics of five network topologies.
arXiv Detail & Related papers (2023-09-08T16:41:08Z) - Higher-order mutual information reveals synergistic sub-networks for
multi-neuron importance [0.483420384410068]
Previous work primarily attributed importance to individual neurons.
In this work, we study which groups of neurons contain synergistic or redundant information.
Results suggest our method can be used for pruning and unsupervised representation learning.
arXiv Detail & Related papers (2022-11-01T12:21:15Z) - Critical Learning Periods for Multisensory Integration in Deep Networks [112.40005682521638]
We show that the ability of a neural network to integrate information from diverse sources hinges critically on being exposed to properly correlated signals during the early phases of training.
We show that critical periods arise from the complex and unstable early transient dynamics, which are decisive of final performance of the trained system and their learned representations.
arXiv Detail & Related papers (2022-10-06T23:50:38Z) - Improving Systematic Generalization Through Modularity and Augmentation [1.2183405753834562]
We investigate how two well-known modeling principles -- modularity and data augmentation -- affect systematic generalization of neural networks.
We show that even in the controlled setting of a synthetic benchmark, achieving systematic generalization remains very difficult.
arXiv Detail & Related papers (2022-02-22T09:04:35Z) - Data-driven emergence of convolutional structure in neural networks [83.4920717252233]
We show how fully-connected neural networks solving a discrimination task can learn a convolutional structure directly from their inputs.
By carefully designing data models, we show that the emergence of this pattern is triggered by the non-Gaussian, higher-order local structure of the inputs.
arXiv Detail & Related papers (2022-02-01T17:11:13Z) - A neural anisotropic view of underspecification in deep learning [60.119023683371736]
We show that the way neural networks handle the underspecification of problems is highly dependent on the data representation.
Our results highlight that understanding the architectural inductive bias in deep learning is fundamental to address the fairness, robustness, and generalization of these systems.
arXiv Detail & Related papers (2021-04-29T14:31:09Z) - Explainable artificial intelligence for mechanics: physics-informing
neural networks for constitutive models [0.0]
In mechanics, the new and active field of physics-informed neural networks attempts to mitigate this disadvantage by designing deep neural networks on the basis of mechanical knowledge.
We propose a first step towards a physics-forming-in approach, which explains neural networks trained on mechanical data a posteriori.
Therein, the principal component analysis decorrelates the distributed representations in cell states of RNNs and allows the comparison to known and fundamental functions.
arXiv Detail & Related papers (2021-04-20T18:38:52Z) - Learning Contact Dynamics using Physically Structured Neural Networks [81.73947303886753]
We use connections between deep neural networks and differential equations to design a family of deep network architectures for representing contact dynamics between objects.
We show that these networks can learn discontinuous contact events in a data-efficient manner from noisy observations.
Our results indicate that an idealised form of touch feedback is a key component of making this learning problem tractable.
arXiv Detail & Related papers (2021-02-22T17:33:51Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.