Critical Learning Periods for Multisensory Integration in Deep Networks

• URL: http://arxiv.org/abs/2210.04643v2
• Date: Thu, 14 Sep 2023 21:48:09 GMT
• Title: Critical Learning Periods for Multisensory Integration in Deep Networks
• Authors: Michael Kleinman, Alessandro Achille, Stefano Soatto
• Abstract summary: We show that the ability of a neural network to integrate information from diverse sources hinges critically on being exposed to properly correlated signals during the early phases of training. We show that critical periods arise from the complex and unstable early transient dynamics, which are decisive of final performance of the trained system and their learned representations.
• Score: 112.40005682521638
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We show that the ability of a neural network to integrate information from diverse sources hinges critically on being exposed to properly correlated signals during the early phases of training. Interfering with the learning process during this initial stage can permanently impair the development of a skill, both in artificial and biological systems where the phenomenon is known as a critical learning period. We show that critical periods arise from the complex and unstable early transient dynamics, which are decisive of final performance of the trained system and their learned representations. This evidence challenges the view, engendered by analysis of wide and shallow networks, that early learning dynamics of neural networks are simple, akin to those of a linear model. Indeed, we show that even deep linear networks exhibit critical learning periods for multi-source integration, while shallow networks do not. To better understand how the internal representations change according to disturbances or sensory deficits, we introduce a new measure of source sensitivity, which allows us to track the inhibition and integration of sources during training. Our analysis of inhibition suggests cross-source reconstruction as a natural auxiliary training objective, and indeed we show that architectures trained with cross-sensor reconstruction objectives are remarkably more resilient to critical periods. Our findings suggest that the recent success in self-supervised multi-modal training compared to previous supervised efforts may be in part due to more robust learning dynamics and not solely due to better architectures and/or more data.

Related papers

• Brain-inspired continual pre-trained learner via silent synaptic consolidation [2.872028467114491]
  Artsy is inspired by the activation mechanisms of silent synapses via spike-timing-dependent plasticity observed in mature brains. It mimics mature brain dynamics by maintaining memory stability for previously learned knowledge within the pre-trained network. During inference, artificial silent and functional synapses are utilized to establish precise connections between the pre-trained network and the sub-networks.
  arXiv  Detail & Related papers  (2024-10-08T10:56:19Z)
• How connectivity structure shapes rich and lazy learning in neural circuits [14.236853424595333]
  We investigate how the structure of the initial weights -- in particular their effective rank -- influences the network learning regime. Our research highlights the pivotal role of initial weight structures in shaping learning regimes.
  arXiv  Detail & Related papers  (2023-10-12T17:08:45Z)
• Critical Learning Periods Emerge Even in Deep Linear Networks [102.89011295243334]
  Critical learning periods are periods early in development where temporary sensory deficits can have a permanent effect on behavior and learned representations. Despite the radical differences between biological and artificial networks, critical learning periods have been empirically observed in both systems.
  arXiv  Detail & Related papers  (2023-08-23T16:01:50Z)
• Synergistic information supports modality integration and flexible learning in neural networks solving multiple tasks [107.8565143456161]
  We investigate the information processing strategies adopted by simple artificial neural networks performing a variety of cognitive tasks. Results show that synergy increases as neural networks learn multiple diverse tasks. randomly turning off neurons during training through dropout increases network redundancy, corresponding to an increase in robustness.
  arXiv  Detail & Related papers  (2022-10-06T15:36:27Z)
• Data-driven emergence of convolutional structure in neural networks [83.4920717252233]
  We show how fully-connected neural networks solving a discrimination task can learn a convolutional structure directly from their inputs. By carefully designing data models, we show that the emergence of this pattern is triggered by the non-Gaussian, higher-order local structure of the inputs.
  arXiv  Detail & Related papers  (2022-02-01T17:11:13Z)
• Statistical Mechanical Analysis of Catastrophic Forgetting in Continual Learning with Teacher and Student Networks [5.209145866174911]
  When a computational system continuously learns from an ever-changing environment, it rapidly forgets its past experiences. We provide the theoretical framework for analyzing catastrophic forgetting by using teacher-student learning. We find that the network can avoid catastrophic forgetting when the similarity among input distributions is small and that of the input-output relationship of the target functions is large.
  arXiv  Detail & Related papers  (2021-05-16T09:02:48Z)
• Learning Contact Dynamics using Physically Structured Neural Networks [81.73947303886753]
  We use connections between deep neural networks and differential equations to design a family of deep network architectures for representing contact dynamics between objects. We show that these networks can learn discontinuous contact events in a data-efficient manner from noisy observations. Our results indicate that an idealised form of touch feedback is a key component of making this learning problem tractable.
  arXiv  Detail & Related papers  (2021-02-22T17:33:51Z)
• The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networks [43.860358308049044]
  In work, we show that these common perceptions can be completely false in the early phase of learning. We argue that this surprising simplicity can persist in networks with more layers with convolutional architecture.
  arXiv  Detail & Related papers  (2020-06-25T17:42:49Z)
• Understanding the Role of Training Regimes in Continual Learning [51.32945003239048]
  Catastrophic forgetting affects the training of neural networks, limiting their ability to learn multiple tasks sequentially. We study the effect of dropout, learning rate decay, and batch size, on forming training regimes that widen the tasks' local minima.
  arXiv  Detail & Related papers  (2020-06-12T06:00:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.