Algorithmic insights on continual learning from fruit flies
- URL: http://arxiv.org/abs/2107.07617v1
- Date: Thu, 15 Jul 2021 21:28:53 GMT
- Title: Algorithmic insights on continual learning from fruit flies
- Authors: Yang Shen, Sanjoy Dasgupta, Saket Navlakha
- Abstract summary: Continual learning in computational systems is challenging due to catastrophic forgetting.
We discovered a two layer neural circuit in the fruit fly olfactory system that addresses this challenge.
In the first layer, odors are encoded using sparse, high dimensional representations, which reduces memory interference.
In the second layer, only the synapses between odor activated neurons and the output neuron associated with the odor are modified during learning.
- Score: 22.34773145953582
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Continual learning in computational systems is challenging due to
catastrophic forgetting. We discovered a two layer neural circuit in the fruit
fly olfactory system that addresses this challenge by uniquely combining sparse
coding and associative learning. In the first layer, odors are encoded using
sparse, high dimensional representations, which reduces memory interference by
activating non overlapping populations of neurons for different odors. In the
second layer, only the synapses between odor activated neurons and the output
neuron associated with the odor are modified during learning; the rest of the
weights are frozen to prevent unrelated memories from being overwritten. We
show empirically and analytically that this simple and lightweight algorithm
significantly boosts continual learning performance. The fly associative
learning algorithm is strikingly similar to the classic perceptron learning
algorithm, albeit two modifications, which we show are critical for reducing
catastrophic forgetting. Overall, fruit flies evolved an efficient lifelong
learning algorithm, and circuit mechanisms from neuroscience can be translated
to improve machine computation.
Related papers
- Hebbian Learning based Orthogonal Projection for Continual Learning of
Spiking Neural Networks [74.3099028063756]
We develop a new method with neuronal operations based on lateral connections and Hebbian learning.
We show that Hebbian and anti-Hebbian learning on recurrent lateral connections can effectively extract the principal subspace of neural activities.
Our method consistently solves for spiking neural networks with nearly zero forgetting.
arXiv Detail & Related papers (2024-02-19T09:29:37Z) - Spiking mode-based neural networks [2.5690340428649328]
Spiking neural networks play an important role in brain-like neuromorphic computations and in studying working mechanisms of neural circuits.
One drawback of training a large scale spiking neural network is that updating all weights is quite expensive.
We propose a spiking mode-based training protocol, where the recurrent weight matrix is explained as a Hopfield-like multiplication of three matrices.
arXiv Detail & Related papers (2023-10-23T06:54:17Z) - Unsupervised Learning of Invariance Transformations [105.54048699217668]
We develop an algorithmic framework for finding approximate graph automorphisms.
We discuss how this framework can be used to find approximate automorphisms in weighted graphs in general.
arXiv Detail & Related papers (2023-07-24T17:03:28Z) - Spiking neural network for nonlinear regression [68.8204255655161]
Spiking neural networks carry the potential for a massive reduction in memory and energy consumption.
They introduce temporal and neuronal sparsity, which can be exploited by next-generation neuromorphic hardware.
A framework for regression using spiking neural networks is proposed.
arXiv Detail & Related papers (2022-10-06T13:04:45Z) - The Role Of Biology In Deep Learning [0.0]
This work summarizes some of that history and incorporates modern theoretical neuroscience into experiments with artificial neural networks from the field of deep learning.
Specifically, iterative magnitude pruning is used to train sparsely connected networks with 33x fewer weights without loss in performance.
These are used to test and ultimately reject the hypothesis that weight sparsity alone improves image noise robustness.
arXiv Detail & Related papers (2022-09-07T23:22:57Z) - Some thoughts on catastrophic forgetting and how to learn an algorithm [0.0]
We propose to use a neural network with a different architecture that can be trained to recover the correct algorithm for the addition of binary numbers.
The neural network not only does not suffer from catastrophic forgetting but it improves its predictive power on unseen pairs of numbers as training progresses.
arXiv Detail & Related papers (2021-08-09T11:12:43Z) - Artificial Neural Variability for Deep Learning: On Overfitting, Noise
Memorization, and Catastrophic Forgetting [135.0863818867184]
artificial neural variability (ANV) helps artificial neural networks learn some advantages from natural'' neural networks.
ANV plays as an implicit regularizer of the mutual information between the training data and the learned model.
It can effectively relieve overfitting, label noise memorization, and catastrophic forgetting at negligible costs.
arXiv Detail & Related papers (2020-11-12T06:06:33Z) - Faster Biological Gradient Descent Learning [0.0]
Back-propagation is a popular machine learning algorithm that uses gradient descent in training neural networks for supervised learning.
We have come up with a simple and local gradient descent optimization algorithm that can reduce training time.
Our algorithm is found to speed up learning, particularly for small networks.
arXiv Detail & Related papers (2020-09-27T05:26:56Z) - Learning compositional functions via multiplicative weight updates [97.9457834009578]
We show that multiplicative weight updates satisfy a descent lemma tailored to compositional functions.
We show that Madam can train state of the art neural network architectures without learning rate tuning.
arXiv Detail & Related papers (2020-06-25T17:05:19Z) - Towards Efficient Processing and Learning with Spikes: New Approaches
for Multi-Spike Learning [59.249322621035056]
We propose two new multi-spike learning rules which demonstrate better performance over other baselines on various tasks.
In the feature detection task, we re-examine the ability of unsupervised STDP with its limitations being presented.
Our proposed learning rules can reliably solve the task over a wide range of conditions without specific constraints being applied.
arXiv Detail & Related papers (2020-05-02T06:41:20Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.