Encoding-based Memory Modules for Recurrent Neural Networks
- URL: http://arxiv.org/abs/2001.11771v1
- Date: Fri, 31 Jan 2020 11:14:27 GMT
- Title: Encoding-based Memory Modules for Recurrent Neural Networks
- Authors: Antonio Carta, Alessandro Sperduti, Davide Bacciu
- Abstract summary: We study the memorization subtask from the point of view of the design and training of recurrent neural networks.
We propose a new model, the Linear Memory Network, which features an encoding-based memorization component built with a linear autoencoder for sequences.
- Score: 79.42778415729475
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Learning to solve sequential tasks with recurrent models requires the ability
to memorize long sequences and to extract task-relevant features from them. In
this paper, we study the memorization subtask from the point of view of the
design and training of recurrent neural networks. We propose a new model, the
Linear Memory Network, which features an encoding-based memorization component
built with a linear autoencoder for sequences. We extend the memorization
component with a modular memory that encodes the hidden state sequence at
different sampling frequencies. Additionally, we provide a specialized training
algorithm that initializes the memory to efficiently encode the hidden
activations of the network. The experimental results on synthetic and
real-world datasets show that specializing the training algorithm to train the
memorization component always improves the final performance whenever the
memorization of long sequences is necessary to solve the problem.
Related papers
- Generalizability of Memorization Neural Networks [13.144557876007358]
memorization is widely believed to have a close relationship with the strong generalizability of deep learning.
We show that, in order for the memorization networks to be generalizable, the width of the network must be at least equal to the dimension of the data.
It is also shown that there exist data distributions such that, to be generalizable for them, the memorization network must have an exponential number of parameters in the data dimension.
arXiv Detail & Related papers (2024-11-01T05:18:46Z) - Sequential Learning in the Dense Associative Memory [1.2289361708127877]
We investigate the performance of the Dense Associative Memory in sequential learning problems.
We show that existing sequential learning methods can be applied to the Dense Associative Memory to improve sequential learning performance.
arXiv Detail & Related papers (2024-09-24T04:23:00Z) - Enhancing Length Extrapolation in Sequential Models with Pointer-Augmented Neural Memory [66.88278207591294]
We propose Pointer-Augmented Neural Memory (PANM) to help neural networks understand and apply symbol processing to new, longer sequences of data.
PANM integrates an external neural memory that uses novel physical addresses and pointer manipulation techniques to mimic human and computer symbol processing abilities.
arXiv Detail & Related papers (2024-04-18T03:03:46Z) - Learning Sequence Attractors in Recurrent Networks with Hidden Neurons [8.639486652067024]
We show that to store arbitrary pattern sequences, it is necessary for the network to include hidden neurons.
We develop a local learning algorithm to learn sequence attractors in the networks with hidden neurons.
We demonstrate that the network model can store and retrieve sequences robustly on synthetic and real-world datasets.
arXiv Detail & Related papers (2024-04-03T13:29:12Z) - The Curious Case of Benign Memorization [19.74244993871716]
We show that under training protocols that include data augmentation, neural networks learn to memorize entirely random labels in a benign way.
We demonstrate that deep models have the surprising ability to separate noise from signal by distributing the task of memorization and feature learning to different layers.
arXiv Detail & Related papers (2022-10-25T13:41:31Z) - Measures of Information Reflect Memorization Patterns [53.71420125627608]
We show that the diversity in the activation patterns of different neurons is reflective of model generalization and memorization.
Importantly, we discover that information organization points to the two forms of memorization, even for neural activations computed on unlabelled in-distribution examples.
arXiv Detail & Related papers (2022-10-17T20:15:24Z) - Learning to Rehearse in Long Sequence Memorization [107.14601197043308]
Existing reasoning tasks often have an important assumption that the input contents can be always accessed while reasoning.
Memory augmented neural networks introduce a human-like write-read memory to compress and memorize the long input sequence in one pass.
But they have two serious drawbacks: 1) they continually update the memory from current information and inevitably forget the early contents; 2) they do not distinguish what information is important and treat all contents equally.
We propose the Rehearsal Memory to enhance long-sequence memorization by self-supervised rehearsal with a history sampler.
arXiv Detail & Related papers (2021-06-02T11:58:30Z) - Reservoir Stack Machines [77.12475691708838]
Memory-augmented neural networks equip a recurrent neural network with an explicit memory to support tasks that require information storage.
We introduce the reservoir stack machine, a model which can provably recognize all deterministic context-free languages.
Our results show that the reservoir stack machine achieves zero error, even on test sequences longer than the training data.
arXiv Detail & Related papers (2021-05-04T16:50:40Z) - Incremental Training of a Recurrent Neural Network Exploiting a
Multi-Scale Dynamic Memory [79.42778415729475]
We propose a novel incrementally trained recurrent architecture targeting explicitly multi-scale learning.
We show how to extend the architecture of a simple RNN by separating its hidden state into different modules.
We discuss a training algorithm where new modules are iteratively added to the model to learn progressively longer dependencies.
arXiv Detail & Related papers (2020-06-29T08:35:49Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.