Neuromorphic Algorithm-hardware Codesign for Temporal Pattern Learning
- URL: http://arxiv.org/abs/2104.10712v2
- Date: Fri, 7 May 2021 03:41:52 GMT
- Title: Neuromorphic Algorithm-hardware Codesign for Temporal Pattern Learning
- Authors: Haowen Fang, Brady Taylor, Ziru Li, Zaidao Mei, Hai Li, Qinru Qiu
- Abstract summary: We derive an efficient training algorithm for Leaky Integrate and Fire neurons, which is capable of training a SNN to learn complex spatial temporal patterns.
We have developed a CMOS circuit implementation for a memristor-based network of neuron and synapses which retains critical neural dynamics with reduced complexity.
- Score: 11.781094547718595
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Neuromorphic computing and spiking neural networks (SNN) mimic the behavior
of biological systems and have drawn interest for their potential to perform
cognitive tasks with high energy efficiency. However, some factors such as
temporal dynamics and spike timings prove critical for information processing
but are often ignored by existing works, limiting the performance and
applications of neuromorphic computing. On one hand, due to the lack of
effective SNN training algorithms, it is difficult to utilize the temporal
neural dynamics. Many existing algorithms still treat neuron activation
statistically. On the other hand, utilizing temporal neural dynamics also poses
challenges to hardware design. Synapses exhibit temporal dynamics, serving as
memory units that hold historical information, but are often simplified as a
connection with weight. Most current models integrate synaptic activations in
some storage medium to represent membrane potential and institute a hard reset
of membrane potential after the neuron emits a spike. This is done for its
simplicity in hardware, requiring only a "clear" signal to wipe the storage
medium, but destroys temporal information stored in the neuron.
In this work, we derive an efficient training algorithm for Leaky Integrate
and Fire neurons, which is capable of training a SNN to learn complex spatial
temporal patterns. We achieved competitive accuracy on two complex datasets. We
also demonstrate the advantage of our model by a novel temporal pattern
association task. Codesigned with this algorithm, we have developed a CMOS
circuit implementation for a memristor-based network of neuron and synapses
which retains critical neural dynamics with reduced complexity. This circuit
implementation of the neuron model is simulated to demonstrate its ability to
react to temporal spiking patterns with an adaptive threshold.
Related papers
- Temporal Spiking Neural Networks with Synaptic Delay for Graph Reasoning [91.29876772547348]
Spiking neural networks (SNNs) are investigated as biologically inspired models of neural computation.
This paper reveals that SNNs, when amalgamated with synaptic delay and temporal coding, are proficient in executing (knowledge) graph reasoning.
arXiv Detail & Related papers (2024-05-27T05:53:30Z) - Order from chaos: Interplay of development and learning in recurrent
networks of structured neurons [1.6880888629604525]
We introduce a fully local, always-on plasticity rule to learn complex sequences in a recurrent network comprised of two populations.
Our model is resource-efficient, enabling the learning of complex sequences using only a small number of neurons.
We demonstrate these features in a mock-up of birdsong learning, in which our networks first learn a long, non-Markovian sequence.
arXiv Detail & Related papers (2024-02-26T17:30:34Z) - The Expressive Leaky Memory Neuron: an Efficient and Expressive Phenomenological Neuron Model Can Solve Long-Horizon Tasks [64.08042492426992]
We introduce the Expressive Memory (ELM) neuron model, a biologically inspired model of a cortical neuron.
Our ELM neuron can accurately match the aforementioned input-output relationship with under ten thousand trainable parameters.
We evaluate it on various tasks with demanding temporal structures, including the Long Range Arena (LRA) datasets.
arXiv Detail & Related papers (2023-06-14T13:34:13Z) - Spiking neural network for nonlinear regression [68.8204255655161]
Spiking neural networks carry the potential for a massive reduction in memory and energy consumption.
They introduce temporal and neuronal sparsity, which can be exploited by next-generation neuromorphic hardware.
A framework for regression using spiking neural networks is proposed.
arXiv Detail & Related papers (2022-10-06T13:04:45Z) - Learning to Modulate Random Weights: Neuromodulation-inspired Neural
Networks For Efficient Continual Learning [1.9580473532948401]
We introduce a novel neural network architecture inspired by neuromodulation in biological nervous systems.
We show that this approach has strong learning performance per task despite the very small number of learnable parameters.
arXiv Detail & Related papers (2022-04-08T21:12:13Z) - Reducing Catastrophic Forgetting in Self Organizing Maps with
Internally-Induced Generative Replay [67.50637511633212]
A lifelong learning agent is able to continually learn from potentially infinite streams of pattern sensory data.
One major historic difficulty in building agents that adapt is that neural systems struggle to retain previously-acquired knowledge when learning from new samples.
This problem is known as catastrophic forgetting (interference) and remains an unsolved problem in the domain of machine learning to this day.
arXiv Detail & Related papers (2021-12-09T07:11:14Z) - Mapping and Validating a Point Neuron Model on Intel's Neuromorphic
Hardware Loihi [77.34726150561087]
We investigate the potential of Intel's fifth generation neuromorphic chip - Loihi'
Loihi is based on the novel idea of Spiking Neural Networks (SNNs) emulating the neurons in the brain.
We find that Loihi replicates classical simulations very efficiently and scales notably well in terms of both time and energy performance as the networks get larger.
arXiv Detail & Related papers (2021-09-22T16:52:51Z) - Astrocytes mediate analogous memory in a multi-layer neuron-astrocytic
network [52.77024349608834]
We show how a piece of information can be maintained as a robust activity pattern for several seconds then completely disappear if no other stimuli come.
This kind of short-term memory can keep operative information for seconds, then completely forget it to avoid overlapping with forthcoming patterns.
We show how arbitrary patterns can be loaded, then stored for a certain interval of time, and retrieved if the appropriate clue pattern is applied to the input.
arXiv Detail & Related papers (2021-08-31T16:13:15Z) - A Deep 2-Dimensional Dynamical Spiking Neuronal Network for Temporal
Encoding trained with STDP [10.982390333064536]
We show that a large, deep layered SNN with dynamical, chaotic activity mimicking the mammalian cortex is capable of encoding information from temporal data.
We argue that the randomness inherent in the network weights allow the neurons to form groups that encode the temporal data being inputted after self-organizing with STDP.
We analyze the network in terms of network entropy as a metric of information transfer.
arXiv Detail & Related papers (2020-09-01T17:12:18Z) - Exploiting Neuron and Synapse Filter Dynamics in Spatial Temporal
Learning of Deep Spiking Neural Network [7.503685643036081]
A bio-plausible SNN model with spatial-temporal property is a complex dynamic system.
We formulate SNN as a network of infinite impulse response (IIR) filters with neuron nonlinearity.
We propose a training algorithm that is capable to learn spatial-temporal patterns by searching for the optimal synapse filter kernels and weights.
arXiv Detail & Related papers (2020-02-19T01:27:39Z) - Structural plasticity on an accelerated analog neuromorphic hardware
system [0.46180371154032884]
We present a strategy to achieve structural plasticity by constantly rewiring the pre- and gpostsynaptic partners.
We implemented this algorithm on the analog neuromorphic system BrainScaleS-2.
We evaluated our implementation in a simple supervised learning scenario, showing its ability to optimize the network topology.
arXiv Detail & Related papers (2019-12-27T10:15:58Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.