Experimentally realized memristive memory augmented neural network
- URL: http://arxiv.org/abs/2204.07429v1
- Date: Fri, 15 Apr 2022 11:52:30 GMT
- Title: Experimentally realized memristive memory augmented neural network
- Authors: Ruibin Mao (1), Bo Wen (1), Yahui Zhao (1), Arman Kazemi (2 and 3),
Ann Franchesca Laguna (3), Michael Neimier (3), X. Sharon Hu (3), Xia Sheng
(2), Catherine E. Graves (2), John Paul Strachan (4, 5), Can Li (1) ((1) The
University of Hong Kong, (2) Hewlett Packard Labs, (3) University of Notre
Dame, (4) Peter Gr\"unberg Institut (PGI-14), (5) RWTH Aachen University)
- Abstract summary: Lifelong on-device learning is a key challenge for machine intelligence.
Memory augmented neural network has been proposed to achieve the goal, but the memory module has to be stored in an off-chip memory.
We implement the entire memory augmented neural network architecture in a fully integrated memristive crossbar platform.
- Score: 0.0
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Lifelong on-device learning is a key challenge for machine intelligence, and
this requires learning from few, often single, samples. Memory augmented neural
network has been proposed to achieve the goal, but the memory module has to be
stored in an off-chip memory due to its size. Therefore the practical use has
been heavily limited. Previous works on emerging memory-based implementation
have difficulties in scaling up because different modules with various
structures are difficult to integrate on the same chip and the small sense
margin of the content addressable memory for the memory module heavily limited
the degree of mismatch calculation. In this work, we implement the entire
memory augmented neural network architecture in a fully integrated memristive
crossbar platform and achieve an accuracy that closely matches standard
software on digital hardware for the Omniglot dataset. The successful
demonstration is supported by implementing new functions in crossbars in
addition to widely reported matrix multiplications. For example, the
locality-sensitive hashing operation is implemented in crossbar arrays by
exploiting the intrinsic stochasticity of memristor devices. Besides, the
content-addressable memory module is realized in crossbars, which also supports
the degree of mismatches. Simulations based on experimentally validated models
show such an implementation can be efficiently scaled up for one-shot learning
on the Mini-ImageNet dataset. The successful demonstration paves the way for
practical on-device lifelong learning and opens possibilities for novel
attention-based algorithms not possible in conventional hardware.
Related papers
- Heterogenous Memory Augmented Neural Networks [84.29338268789684]
We introduce a novel heterogeneous memory augmentation approach for neural networks.
By introducing learnable memory tokens with attention mechanism, we can effectively boost performance without huge computational overhead.
We show our approach on various image and graph-based tasks under both in-distribution (ID) and out-of-distribution (OOD) conditions.
arXiv Detail & Related papers (2023-10-17T01:05:28Z) - Pex: Memory-efficient Microcontroller Deep Learning through Partial
Execution [11.336229510791481]
We discuss a novel execution paradigm for microcontroller deep learning.
It modifies the execution of neural networks to avoid materialising full buffers in memory.
This is achieved by exploiting the properties of operators, which can consume/produce a fraction of their input/output at a time.
arXiv Detail & Related papers (2022-11-30T18:47:30Z) - A Co-design view of Compute in-Memory with Non-Volatile Elements for
Neural Networks [12.042322495445196]
We discuss how compute-in-memory can play an important part in the next generation of computing hardware.
A non-volatile memory based cross-bar architecture forms the heart of an engine that uses an analog process to parallelize the matrix vector multiplication operation.
The cross-bar architecture, at times referred to as a neuromorphic approach, can be a key hardware element in future computing machines.
arXiv Detail & Related papers (2022-06-03T15:59:46Z) - Memory-Guided Semantic Learning Network for Temporal Sentence Grounding [55.31041933103645]
We propose a memory-augmented network that learns and memorizes the rarely appeared content in TSG tasks.
MGSL-Net consists of three main parts: a cross-modal inter-action module, a memory augmentation module, and a heterogeneous attention module.
arXiv Detail & Related papers (2022-01-03T02:32:06Z) - MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learning [72.80896338009579]
We find that the memory bottleneck is due to the imbalanced memory distribution in convolutional neural network (CNN) designs.
We propose a generic patch-by-patch inference scheduling, which significantly cuts down the peak memory.
We automate the process with neural architecture search to jointly optimize the neural architecture and inference scheduling, leading to MCUNetV2.
arXiv Detail & Related papers (2021-10-28T17:58:45Z) - Recognition of handwritten MNIST digits on low-memory 2 Kb RAM Arduino
board using LogNNet reservoir neural network [0.0]
The presented algorithm for recognizing handwritten digits of the MNIST database, created on the LogNNet reservoir neural network, reaches the recognition accuracy of 82%.
The simple structure of the algorithm, with appropriate training, can be adapted for wide practical application, for example, for creating mobile biosensors for early diagnosis of adverse events in medicine.
arXiv Detail & Related papers (2021-04-20T18:16:23Z) - Robust High-dimensional Memory-augmented Neural Networks [13.82206983716435]
Memory-augmented neural networks enhance neural networks with an explicit memory to overcome these issues.
Access to this explicit memory occurs via soft read and write operations involving every individual memory entry.
We propose a robust architecture that employs a computational memory unit as the explicit memory performing analog in-memory computation on high-dimensional (HD) vectors.
arXiv Detail & Related papers (2020-10-05T12:01:56Z) - Diagonal Memory Optimisation for Machine Learning on Micro-controllers [21.222568055417717]
Micro controllers and low power CPUs are increasingly being used to perform inference with machine learning models.
Small amounts of RAM available on these targets sets limits on size of models which can be executed.
diagonal memory optimisation technique is described and shown to achieve memory savings of up to 34.5% when applied to eleven common models.
arXiv Detail & Related papers (2020-10-04T19:45:55Z) - Reservoir Memory Machines as Neural Computers [70.5993855765376]
Differentiable neural computers extend artificial neural networks with an explicit memory without interference.
We achieve some of the computational capabilities of differentiable neural computers with a model that can be trained very efficiently.
arXiv Detail & Related papers (2020-09-14T12:01:30Z) - Incremental Training of a Recurrent Neural Network Exploiting a
Multi-Scale Dynamic Memory [79.42778415729475]
We propose a novel incrementally trained recurrent architecture targeting explicitly multi-scale learning.
We show how to extend the architecture of a simple RNN by separating its hidden state into different modules.
We discuss a training algorithm where new modules are iteratively added to the model to learn progressively longer dependencies.
arXiv Detail & Related papers (2020-06-29T08:35:49Z) - One-step regression and classification with crosspoint resistive memory
arrays [62.997667081978825]
High speed, low energy computing machines are in demand to enable real-time artificial intelligence at the edge.
One-step learning is supported by simulations of the prediction of the cost of a house in Boston and the training of a 2-layer neural network for MNIST digit recognition.
Results are all obtained in one computational step, thanks to the physical, parallel, and analog computing within the crosspoint array.
arXiv Detail & Related papers (2020-05-05T08:00:07Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.