Local plasticity rules can learn deep representations using self-supervised contrastive predictions

• URL: http://arxiv.org/abs/2010.08262v5
• Date: Mon, 25 Oct 2021 10:23:56 GMT
• Title: Local plasticity rules can learn deep representations using self-supervised contrastive predictions
• Authors: Bernd Illing, Jean Ventura, Guillaume Bellec, Wulfram Gerstner
• Abstract summary: Learning rules that respect biological constraints, yet yield deep hierarchical representations are still unknown. We propose a learning rule that takes inspiration from neuroscience and recent advances in self-supervised deep learning. We find that networks trained with this self-supervised and local rule build deep hierarchical representations of images, speech and video.
• Score: 3.6868085124383616
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Learning in the brain is poorly understood and learning rules that respect biological constraints, yet yield deep hierarchical representations, are still unknown. Here, we propose a learning rule that takes inspiration from neuroscience and recent advances in self-supervised deep learning. Learning minimizes a simple layer-specific loss function and does not need to back-propagate error signals within or between layers. Instead, weight updates follow a local, Hebbian, learning rule that only depends on pre- and post-synaptic neuronal activity, predictive dendritic input and widely broadcasted modulation factors which are identical for large groups of neurons. The learning rule applies contrastive predictive learning to a causal, biological setting using saccades (i.e. rapid shifts in gaze direction). We find that networks trained with this self-supervised and local rule build deep hierarchical representations of images, speech and video.

Related papers

• NeuroExplainer: Fine-Grained Attention Decoding to Uncover Cortical Development Patterns of Preterm Infants [73.85768093666582]
  We propose an explainable geometric deep network dubbed NeuroExplainer. NeuroExplainer is used to uncover altered infant cortical development patterns associated with preterm birth.
  arXiv  Detail & Related papers  (2023-01-01T12:48:12Z)
• The Influence of Learning Rule on Representation Dynamics in Wide Neural Networks [18.27510863075184]
  We analyze infinite-width deep gradient networks trained with feedback alignment (FA), direct feedback alignment (DFA), and error modulated Hebbian learning (Hebb) We show that, for each of these learning rules, the evolution of the output function at infinite width is governed by a time varying effective neural tangent kernel (eNTK) In the lazy training limit, this eNTK is static and does not evolve, while in the rich mean-field regime this kernel's evolution can be determined self-consistently with dynamical mean field theory (DMFT)
  arXiv  Detail & Related papers  (2022-10-05T11:33:40Z)
• The least-control principle for learning at equilibrium [65.2998274413952]
  We present a new principle for learning equilibrium recurrent neural networks, deep equilibrium models, or meta-learning. Our results shed light on how the brain might learn and offer new ways of approaching a broad class of machine learning problems.
  arXiv  Detail & Related papers  (2022-07-04T11:27:08Z)
• Hebbian Continual Representation Learning [9.54473759331265]
  Continual Learning aims to bring machine learning into a more realistic scenario. We investigate whether biologically inspired Hebbian learning is useful for tackling continual challenges.
  arXiv  Detail & Related papers  (2022-06-28T09:21:03Z)
• Distinguishing Learning Rules with Brain Machine Interfaces [0.0]
  We consider biologically plausible supervised- and reinforcement-learning rules. We derive a metric to distinguish between learning rules by observing changes in the network activity during learning.
  arXiv  Detail & Related papers  (2022-06-27T16:58:30Z)
• Minimizing Control for Credit Assignment with Strong Feedback [65.59995261310529]
  Current methods for gradient-based credit assignment in deep neural networks need infinitesimally small feedback signals. We combine strong feedback influences on neural activity with gradient-based learning and show that this naturally leads to a novel view on neural network optimization. We show that the use of strong feedback in DFC allows learning forward and feedback connections simultaneously, using a learning rule fully local in space and time.
  arXiv  Detail & Related papers  (2022-04-14T22:06:21Z)
• Continual learning under domain transfer with sparse synaptic bursting [2.314558204145174]
  We introduce a system that can learn sequentially over previously unseen datasets with little forgetting over time. We find that our method learns continually under domain transfer with sparse bursts of activity in weights that are recycled across tasks.
  arXiv  Detail & Related papers  (2021-08-26T22:53:27Z)
• PyTorch-Hebbian: facilitating local learning in a deep learning framework [67.67299394613426]
  Hebbian local learning has shown potential as an alternative training mechanism to backpropagation. We propose a framework for thorough and systematic evaluation of local learning rules in existing deep learning pipelines. The framework is used to expand the Krotov-Hopfield learning rule to standard convolutional neural networks without sacrificing accuracy.
  arXiv  Detail & Related papers  (2021-01-31T10:53:08Z)
• Artificial Neural Variability for Deep Learning: On Overfitting, Noise Memorization, and Catastrophic Forgetting [135.0863818867184]
  artificial neural variability (ANV) helps artificial neural networks learn some advantages from natural'' neural networks. ANV plays as an implicit regularizer of the mutual information between the training data and the learned model. It can effectively relieve overfitting, label noise memorization, and catastrophic forgetting at negligible costs.
  arXiv  Detail & Related papers  (2020-11-12T06:06:33Z)
• LoCo: Local Contrastive Representation Learning [93.98029899866866]
  We show that by overlapping local blocks stacking on top of each other, we effectively increase the decoder depth and allow upper blocks to implicitly send feedbacks to lower blocks. This simple design closes the performance gap between local learning and end-to-end contrastive learning algorithms for the first time.
  arXiv  Detail & Related papers  (2020-08-04T05:41:29Z)
• Kernelized information bottleneck leads to biologically plausible 3-factor Hebbian learning in deep networks [6.09170287691728]
  We present a family of learning rules that does not suffer from any of these problems. The resulting rules have a 3-factor Hebbian structure. They do not require precise labels; instead, they rely on the similarity between pairs of desired outputs.
  arXiv  Detail & Related papers  (2020-06-12T12:30:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.