Encoding-based Memory Modules for Recurrent Neural Networks
- URL: http://arxiv.org/abs/2001.11771v1
- Date: Fri, 31 Jan 2020 11:14:27 GMT
- Title: Encoding-based Memory Modules for Recurrent Neural Networks
- Authors: Antonio Carta, Alessandro Sperduti, Davide Bacciu
- Abstract summary: We study the memorization subtask from the point of view of the design and training of recurrent neural networks.
We propose a new model, the Linear Memory Network, which features an encoding-based memorization component built with a linear autoencoder for sequences.
- Score: 79.42778415729475
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Learning to solve sequential tasks with recurrent models requires the ability
to memorize long sequences and to extract task-relevant features from them. In
this paper, we study the memorization subtask from the point of view of the
design and training of recurrent neural networks. We propose a new model, the
Linear Memory Network, which features an encoding-based memorization component
built with a linear autoencoder for sequences. We extend the memorization
component with a modular memory that encodes the hidden state sequence at
different sampling frequencies. Additionally, we provide a specialized training
algorithm that initializes the memory to efficiently encode the hidden
activations of the network. The experimental results on synthetic and
real-world datasets show that specializing the training algorithm to train the
memorization component always improves the final performance whenever the
memorization of long sequences is necessary to solve the problem.
Related papers
- Enhancing Length Extrapolation in Sequential Models with Pointer-Augmented Neural Memory [66.88278207591294]
We propose Pointer-Augmented Neural Memory (PANM) to help neural networks understand and apply symbol processing to new, longer sequences of data.
PANM integrates an external neural memory that uses novel physical addresses and pointer manipulation techniques to mimic human and computer symbol processing abilities.
arXiv Detail & Related papers (2024-04-18T03:03:46Z) - Learning Sequence Attractors in Recurrent Networks with Hidden Neurons [8.639486652067024]
We show that to store arbitrary pattern sequences, it is necessary for the network to include hidden neurons.
We develop a local learning algorithm to learn sequence attractors in the networks with hidden neurons.
We demonstrate that the network model can store and retrieve sequences robustly on synthetic and real-world datasets.
arXiv Detail & Related papers (2024-04-03T13:29:12Z) - The Curious Case of Benign Memorization [19.74244993871716]
We show that under training protocols that include data augmentation, neural networks learn to memorize entirely random labels in a benign way.
We demonstrate that deep models have the surprising ability to separate noise from signal by distributing the task of memorization and feature learning to different layers.
arXiv Detail & Related papers (2022-10-25T13:41:31Z) - Measures of Information Reflect Memorization Patterns [53.71420125627608]
We show that the diversity in the activation patterns of different neurons is reflective of model generalization and memorization.
Importantly, we discover that information organization points to the two forms of memorization, even for neural activations computed on unlabelled in-distribution examples.
arXiv Detail & Related papers (2022-10-17T20:15:24Z) - Learning to Rehearse in Long Sequence Memorization [107.14601197043308]
Existing reasoning tasks often have an important assumption that the input contents can be always accessed while reasoning.
Memory augmented neural networks introduce a human-like write-read memory to compress and memorize the long input sequence in one pass.
But they have two serious drawbacks: 1) they continually update the memory from current information and inevitably forget the early contents; 2) they do not distinguish what information is important and treat all contents equally.
We propose the Rehearsal Memory to enhance long-sequence memorization by self-supervised rehearsal with a history sampler.
arXiv Detail & Related papers (2021-06-02T11:58:30Z) - Reservoir Stack Machines [77.12475691708838]
Memory-augmented neural networks equip a recurrent neural network with an explicit memory to support tasks that require information storage.
We introduce the reservoir stack machine, a model which can provably recognize all deterministic context-free languages.
Our results show that the reservoir stack machine achieves zero error, even on test sequences longer than the training data.
arXiv Detail & Related papers (2021-05-04T16:50:40Z) - Incremental Training of a Recurrent Neural Network Exploiting a
Multi-Scale Dynamic Memory [79.42778415729475]
We propose a novel incrementally trained recurrent architecture targeting explicitly multi-scale learning.
We show how to extend the architecture of a simple RNN by separating its hidden state into different modules.
We discuss a training algorithm where new modules are iteratively added to the model to learn progressively longer dependencies.
arXiv Detail & Related papers (2020-06-29T08:35:49Z) - Triple Memory Networks: a Brain-Inspired Method for Continual Learning [35.40452724755021]
A neural network adjusts its parameters when learning a new task, but then fails to conduct the old tasks well.
The brain has a powerful ability to continually learn new experience without catastrophic interference.
Inspired by such brain strategy, we propose a novel approach named triple memory networks (TMNs) for continual learning.
arXiv Detail & Related papers (2020-03-06T11:35:24Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.