PyTorch-Hebbian: facilitating local learning in a deep learning
framework
- URL: http://arxiv.org/abs/2102.00428v1
- Date: Sun, 31 Jan 2021 10:53:08 GMT
- Title: PyTorch-Hebbian: facilitating local learning in a deep learning
framework
- Authors: Jules Talloen, Joni Dambre, Alexander Vandesompele
- Abstract summary: Hebbian local learning has shown potential as an alternative training mechanism to backpropagation.
We propose a framework for thorough and systematic evaluation of local learning rules in existing deep learning pipelines.
The framework is used to expand the Krotov-Hopfield learning rule to standard convolutional neural networks without sacrificing accuracy.
- Score: 67.67299394613426
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: Recently, unsupervised local learning, based on Hebb's idea that change in
synaptic efficacy depends on the activity of the pre- and postsynaptic neuron
only, has shown potential as an alternative training mechanism to
backpropagation. Unfortunately, Hebbian learning remains experimental and
rarely makes it way into standard deep learning frameworks. In this work, we
investigate the potential of Hebbian learning in the context of standard deep
learning workflows. To this end, a framework for thorough and systematic
evaluation of local learning rules in existing deep learning pipelines is
proposed. Using this framework, the potential of Hebbian learned feature
extractors for image classification is illustrated. In particular, the
framework is used to expand the Krotov-Hopfield learning rule to standard
convolutional neural networks without sacrificing accuracy compared to
end-to-end backpropagation. The source code is available at
https://github.com/Joxis/pytorch-hebbian.
Related papers
- The Cascaded Forward Algorithm for Neural Network Training [61.06444586991505]
We propose a new learning framework for neural networks, namely Cascaded Forward (CaFo) algorithm, which does not rely on BP optimization as that in FF.
Unlike FF, our framework directly outputs label distributions at each cascaded block, which does not require generation of additional negative samples.
In our framework each block can be trained independently, so it can be easily deployed into parallel acceleration systems.
arXiv Detail & Related papers (2023-03-17T02:01:11Z) - Activation Learning by Local Competitions [4.441866681085516]
We develop a biology-inspired learning rule that discovers features by local competitions among neurons.
It is demonstrated that the unsupervised features learned by this local learning rule can serve as a pre-training model.
arXiv Detail & Related papers (2022-09-26T10:43:29Z) - Transfer Learning with Deep Tabular Models [66.67017691983182]
We show that upstream data gives tabular neural networks a decisive advantage over GBDT models.
We propose a realistic medical diagnosis benchmark for tabular transfer learning.
We propose a pseudo-feature method for cases where the upstream and downstream feature sets differ.
arXiv Detail & Related papers (2022-06-30T14:24:32Z) - Hebbian Continual Representation Learning [9.54473759331265]
Continual Learning aims to bring machine learning into a more realistic scenario.
We investigate whether biologically inspired Hebbian learning is useful for tackling continual challenges.
arXiv Detail & Related papers (2022-06-28T09:21:03Z) - Hebbian learning with gradients: Hebbian convolutional neural networks
with modern deep learning frameworks [2.7666483899332643]
We show that Hebbian learning in hierarchical, convolutional neural networks can be implemented almost trivially with modern deep learning frameworks.
We build Hebbian convolutional multi-layer networks for object recognition.
arXiv Detail & Related papers (2021-07-04T20:50:49Z) - Local Critic Training for Model-Parallel Learning of Deep Neural
Networks [94.69202357137452]
We propose a novel model-parallel learning method, called local critic training.
We show that the proposed approach successfully decouples the update process of the layer groups for both convolutional neural networks (CNNs) and recurrent neural networks (RNNs)
We also show that trained networks by the proposed method can be used for structural optimization.
arXiv Detail & Related papers (2021-02-03T09:30:45Z) - Training Convolutional Neural Networks With Hebbian Principal Component
Analysis [10.026753669198108]
Hebbian learning can be used for training the lower or the higher layers of a neural network.
We use a nonlinear Hebbian Principal Component Analysis ( HPCA) learning rule, in place of the Hebbian Winner Takes All (HWTA) strategy.
In particular, the HPCA rule is used to train Convolutional Neural Networks in order to extract relevant features from the CIFAR-10 image dataset.
arXiv Detail & Related papers (2020-12-22T18:17:46Z) - LoCo: Local Contrastive Representation Learning [93.98029899866866]
We show that by overlapping local blocks stacking on top of each other, we effectively increase the decoder depth and allow upper blocks to implicitly send feedbacks to lower blocks.
This simple design closes the performance gap between local learning and end-to-end contrastive learning algorithms for the first time.
arXiv Detail & Related papers (2020-08-04T05:41:29Z) - Rethinking Few-Shot Image Classification: a Good Embedding Is All You
Need? [72.00712736992618]
We show that a simple baseline: learning a supervised or self-supervised representation on the meta-training set, outperforms state-of-the-art few-shot learning methods.
An additional boost can be achieved through the use of self-distillation.
We believe that our findings motivate a rethinking of few-shot image classification benchmarks and the associated role of meta-learning algorithms.
arXiv Detail & Related papers (2020-03-25T17:58:42Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.