Reinforcement Learning with Fast and Forgetful Memory

• URL: http://arxiv.org/abs/2310.04128v1
• Date: Fri, 6 Oct 2023 09:56:26 GMT
• Title: Reinforcement Learning with Fast and Forgetful Memory
• Authors: Steven Morad, Ryan Kortvelesy, Stephan Liwicki, Amanda Prorok
• Abstract summary: We introduce Fast and Forgetful Memory, an algorithm-agnostic memory model designed specifically for Reinforcement Learning (RL) Our approach constrains the model search space via strong structural priors inspired by computational psychology. Fast and Forgetful Memory exhibits training speeds two orders of magnitude faster than recurrent neural networks (RNNs)
• Score: 10.087126455388276
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Nearly all real world tasks are inherently partially observable, necessitating the use of memory in Reinforcement Learning (RL). Most model-free approaches summarize the trajectory into a latent Markov state using memory models borrowed from Supervised Learning (SL), even though RL tends to exhibit different training and efficiency characteristics. Addressing this discrepancy, we introduce Fast and Forgetful Memory, an algorithm-agnostic memory model designed specifically for RL. Our approach constrains the model search space via strong structural priors inspired by computational psychology. It is a drop-in replacement for recurrent neural networks (RNNs) in recurrent RL algorithms, achieving greater reward than RNNs across various recurrent benchmarks and algorithms without changing any hyperparameters. Moreover, Fast and Forgetful Memory exhibits training speeds two orders of magnitude faster than RNNs, attributed to its logarithmic time and linear space complexity. Our implementation is available at https://github.com/proroklab/ffm.

Related papers

• RepCNN: Micro-sized, Mighty Models for Wakeword Detection [3.4888176891918654]
  Always-on machine learning models require a very low memory and compute footprint. We show that a small convolutional model can be better trained by first its computation into a larger multi-branched architecture. We show that our always-on wake-word detector model, RepCNN, provides a good trade-off between latency and accuracy during inference.
  arXiv  Detail & Related papers  (2024-06-04T16:14:19Z)
• MSPipe: Efficient Temporal GNN Training via Staleness-Aware Pipeline [8.889825826072512]
  Memory-based Temporal Graph Neural Networks (MTGNNs) are a class of temporal graph neural networks that utilize a node memory module to capture and retain long-term temporal dependencies. Existing optimizations for static GNNs are not directly applicable to MTGNNs due to differences in training paradigm, model architecture, and the absence of a memory module. We propose MSPipe, a general and efficient framework for MTGNNs that maximizes training throughput while maintaining model accuracy.
  arXiv  Detail & Related papers  (2024-02-23T05:57:22Z)
• Towards Memory- and Time-Efficient Backpropagation for Training Spiking Neural Networks [70.75043144299168]
  Spiking Neural Networks (SNNs) are promising energy-efficient models for neuromorphic computing. We propose the Spatial Learning Through Time (SLTT) method that can achieve high performance while greatly improving training efficiency. Our method achieves state-of-the-art accuracy on ImageNet, while the memory cost and training time are reduced by more than 70% and 50%, respectively, compared with BPTT.
  arXiv  Detail & Related papers  (2023-02-28T05:01:01Z)
• OLLA: Decreasing the Memory Usage of Neural Networks by Optimizing the Lifetime and Location of Arrays [6.418232942455968]
  OLLA is an algorithm that optimize the lifetime and memory location of the tensors used to train neural networks. We present several techniques to simplify the encoding of the problem, and enable our approach to scale to the size of state-of-the-art neural networks.
  arXiv  Detail & Related papers  (2022-10-24T02:39:13Z)
• Improving Computational Efficiency in Visual Reinforcement Learning via Stored Embeddings [89.63764845984076]
  We present Stored Embeddings for Efficient Reinforcement Learning (SEER) SEER is a simple modification of existing off-policy deep reinforcement learning methods. We show that SEER does not degrade the performance of RLizable agents while significantly saving computation and memory.
  arXiv  Detail & Related papers  (2021-03-04T08:14:10Z)
• Memformer: A Memory-Augmented Transformer for Sequence Modeling [55.780849185884996]
  We present Memformer, an efficient neural network for sequence modeling. Our model achieves linear time complexity and constant memory space complexity when processing long sequences.
  arXiv  Detail & Related papers  (2020-10-14T09:03:36Z)
• Optimizing Memory Placement using Evolutionary Graph Reinforcement Learning [56.83172249278467]
  We introduce Evolutionary Graph Reinforcement Learning (EGRL), a method designed for large search spaces. We train and validate our approach directly on the Intel NNP-I chip for inference. We additionally achieve 28-78% speed-up compared to the native NNP-I compiler on all three workloads.
  arXiv  Detail & Related papers  (2020-07-14T18:50:12Z)
• Incremental Training of a Recurrent Neural Network Exploiting a Multi-Scale Dynamic Memory [79.42778415729475]
  We propose a novel incrementally trained recurrent architecture targeting explicitly multi-scale learning. We show how to extend the architecture of a simple RNN by separating its hidden state into different modules. We discuss a training algorithm where new modules are iteratively added to the model to learn progressively longer dependencies.
  arXiv  Detail & Related papers  (2020-06-29T08:35:49Z)
• Recognizing Long Grammatical Sequences Using Recurrent Networks Augmented With An External Differentiable Stack [73.48927855855219]
  Recurrent neural networks (RNNs) are a widely used deep architecture for sequence modeling, generation, and prediction. RNNs generalize poorly over very long sequences, which limits their applicability to many important temporal processing and time series forecasting problems. One way to address these shortcomings is to couple an RNN with an external, differentiable memory structure, such as a stack. In this paper, we improve the memory-augmented RNN with important architectural and state updating mechanisms.
  arXiv  Detail & Related papers  (2020-04-04T14:19:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.