Brain-inspired continual pre-trained learner via silent synaptic consolidation

• URL: http://arxiv.org/abs/2410.05899v1
• Date: Tue, 8 Oct 2024 10:56:19 GMT
• Title: Brain-inspired continual pre-trained learner via silent synaptic consolidation
• Authors: Xuming Ran, Juntao Yao, Yusong Wang, Mingkun Xu, Dianbo Liu,
• Abstract summary: Artsy is inspired by the activation mechanisms of silent synapses via spike-timing-dependent plasticity observed in mature brains. It mimics mature brain dynamics by maintaining memory stability for previously learned knowledge within the pre-trained network. During inference, artificial silent and functional synapses are utilized to establish precise connections between the pre-trained network and the sub-networks.
• Score: 2.872028467114491
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Pre-trained models have demonstrated impressive generalization capabilities, yet they remain vulnerable to catastrophic forgetting when incrementally trained on new tasks. Existing architecture-based strategies encounter two primary challenges: 1) Integrating a pre-trained network with a trainable sub-network complicates the delicate balance between learning plasticity and memory stability across evolving tasks during learning. 2) The absence of robust interconnections between pre-trained networks and various sub-networks limits the effective retrieval of pertinent information during inference. In this study, we introduce the Artsy, inspired by the activation mechanisms of silent synapses via spike-timing-dependent plasticity observed in mature brains, to enhance the continual learning capabilities of pre-trained models. The Artsy integrates two key components: During training, the Artsy mimics mature brain dynamics by maintaining memory stability for previously learned knowledge within the pre-trained network while simultaneously promoting learning plasticity in task-specific sub-networks. During inference, artificial silent and functional synapses are utilized to establish precise connections between the pre-synaptic neurons in the pre-trained network and the post-synaptic neurons in the sub-networks, facilitated through synaptic consolidation, thereby enabling effective extraction of relevant information from test samples. Comprehensive experimental evaluations reveal that our model significantly outperforms conventional methods on class-incremental learning tasks, while also providing enhanced biological interpretability for architecture-based approaches. Moreover, we propose that the Artsy offers a promising avenue for simulating biological synaptic mechanisms, potentially advancing our understanding of neural plasticity in both artificial and biological systems.

Related papers

• Synergistic pathways of modulation enable robust task packing within neural dynamics [0.0]
  We use recurrent network models to probe the distinctions between two forms of contextual modulation of neural dynamics. We demonstrate distinction between these mechanisms at the level of the neuronal dynamics they induce. These characterizations indicate complementarity and synergy in how these mechanisms act, potentially over multiple time-scales.
  arXiv  Detail & Related papers  (2024-08-02T15:12:01Z)
• Enhancing learning in spiking neural networks through neuronal heterogeneity and neuromodulatory signaling [52.06722364186432]
  We propose a biologically-informed framework for enhancing artificial neural networks (ANNs) Our proposed dual-framework approach highlights the potential of spiking neural networks (SNNs) for emulating diverse spiking behaviors. We outline how the proposed approach integrates brain-inspired compartmental models and task-driven SNNs, bioinspiration and complexity.
  arXiv  Detail & Related papers  (2024-07-05T14:11:28Z)
• Theories of synaptic memory consolidation and intelligent plasticity for continual learning [7.573586022424398]
  synaptic plasticity mechanisms must maintain and evolve an internal state. plasticity algorithms must leverage the internal state to intelligently regulate plasticity at individual synapses.
  arXiv  Detail & Related papers  (2024-05-27T08:13:39Z)
• Exploring neural oscillations during speech perception via surrogate gradient spiking neural networks [59.38765771221084]
  We present a physiologically inspired speech recognition architecture compatible and scalable with deep learning frameworks. We show end-to-end gradient descent training leads to the emergence of neural oscillations in the central spiking neural network. Our findings highlight the crucial inhibitory role of feedback mechanisms, such as spike frequency adaptation and recurrent connections, in regulating and synchronising neural activity to improve recognition performance.
  arXiv  Detail & Related papers  (2024-04-22T09:40:07Z)
• Disentangling the Causes of Plasticity Loss in Neural Networks [55.23250269007988]
  We show that loss of plasticity can be decomposed into multiple independent mechanisms. We show that a combination of layer normalization and weight decay is highly effective at maintaining plasticity in a variety of synthetic nonstationary learning tasks.
  arXiv  Detail & Related papers  (2024-02-29T00:02:33Z)
• A Study of Biologically Plausible Neural Network: The Role and Interactions of Brain-Inspired Mechanisms in Continual Learning [13.041607703862724]
  Humans excel at continually acquiring, consolidating, and retaining information from an ever-changing environment, whereas artificial neural networks (ANNs) exhibit catastrophic forgetting. We consider a biologically plausible framework that constitutes separate populations of exclusively excitatory and inhibitory neurons that adhere to Dale's principle. We then conduct a comprehensive study on the role and interactions of different mechanisms inspired by the brain, including sparse non-overlapping representations, Hebbian learning, synaptic consolidation, and replay of past activations that accompanied the learning event.
  arXiv  Detail & Related papers  (2023-04-13T16:34:12Z)
• Contrastive-Signal-Dependent Plasticity: Self-Supervised Learning in Spiking Neural Circuits [61.94533459151743]
  This work addresses the challenge of designing neurobiologically-motivated schemes for adjusting the synapses of spiking networks. Our experimental simulations demonstrate a consistent advantage over other biologically-plausible approaches when training recurrent spiking networks.
  arXiv  Detail & Related papers  (2023-03-30T02:40:28Z)
• Critical Learning Periods for Multisensory Integration in Deep Networks [112.40005682521638]
  We show that the ability of a neural network to integrate information from diverse sources hinges critically on being exposed to properly correlated signals during the early phases of training. We show that critical periods arise from the complex and unstable early transient dynamics, which are decisive of final performance of the trained system and their learned representations.
  arXiv  Detail & Related papers  (2022-10-06T23:50:38Z)
• Synergistic information supports modality integration and flexible learning in neural networks solving multiple tasks [107.8565143456161]
  We investigate the information processing strategies adopted by simple artificial neural networks performing a variety of cognitive tasks. Results show that synergy increases as neural networks learn multiple diverse tasks. randomly turning off neurons during training through dropout increases network redundancy, corresponding to an increase in robustness.
  arXiv  Detail & Related papers  (2022-10-06T15:36:27Z)
• Embodied Synaptic Plasticity with Online Reinforcement learning [5.6006805285925445]
  This paper contributes to bringing the fields of computational neuroscience and robotics closer together by integrating open-source software components from these two fields. We demonstrate this framework to evaluate Synaptic Plasticity with Online REinforcement learning (SPORE), a reward-learning rule based on synaptic sampling, on two visuomotor tasks.
  arXiv  Detail & Related papers  (2020-03-03T10:29:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.