Reservoir Computing for Fast, Simplified Reinforcement Learning on Memory Tasks

• URL: http://arxiv.org/abs/2412.13093v1
• Date: Tue, 17 Dec 2024 17:02:06 GMT
• Title: Reservoir Computing for Fast, Simplified Reinforcement Learning on Memory Tasks
• Authors: Kevin McKee,
• Abstract summary: Reservoir computing greatly simplifies and speeds up reinforcement learning on memory tasks. In particular, these findings offer significant benefit to meta-learning that depends primarily on efficient and highly general memory systems.
• Score: 0.0
• License:
• Abstract: Tasks in which rewards depend upon past information not available in the current observation set can only be solved by agents that are equipped with short-term memory. Usual choices for memory modules include trainable recurrent hidden layers, often with gated memory. Reservoir computing presents an alternative, in which a recurrent layer is not trained, but rather has a set of fixed, sparse recurrent weights. The weights are scaled to produce stable dynamical behavior such that the reservoir state contains a high-dimensional, nonlinear impulse response function of the inputs. An output decoder network can then be used to map the compressive history represented by the reservoir's state to any outputs, including agent actions or predictions. In this study, we find that reservoir computing greatly simplifies and speeds up reinforcement learning on memory tasks by (1) eliminating the need for backpropagation of gradients through time, (2) presenting all recent history simultaneously to the downstream network, and (3) performing many useful and generic nonlinear computations upstream from the trained modules. In particular, these findings offer significant benefit to meta-learning that depends primarily on efficient and highly general memory systems.

Related papers

• SHERL: Synthesizing High Accuracy and Efficient Memory for Resource-Limited Transfer Learning [63.93193829913252]
  We propose an innovative METL strategy called SHERL for resource-limited scenarios. In the early route, intermediate outputs are consolidated via an anti-redundancy operation. In the late route, utilizing minimal late pre-trained layers could alleviate the peak demand on memory overhead.
  arXiv  Detail & Related papers  (2024-07-10T10:22:35Z)
• Simple Cycle Reservoirs are Universal [0.358439716487063]
  Reservoir models form a subclass of recurrent neural networks with fixed non-trainable input and dynamic coupling weights. We show that they are capable of universal approximation of any unrestricted linear reservoir system.
  arXiv  Detail & Related papers  (2023-08-21T15:35:59Z)
• Deep Q-network using reservoir computing with multi-layered readout [0.0]
  Recurrent neural network (RNN) based reinforcement learning (RL) is used for learning context-dependent tasks. An approach with replay memory introducing reservoir computing has been proposed, which trains an agent without BPTT. This paper shows that the performance of this method improves by using a multi-layered neural network for the readout layer.
  arXiv  Detail & Related papers  (2022-03-03T00:32:55Z)
• Memory-Guided Semantic Learning Network for Temporal Sentence Grounding [55.31041933103645]
  We propose a memory-augmented network that learns and memorizes the rarely appeared content in TSG tasks. MGSL-Net consists of three main parts: a cross-modal inter-action module, a memory augmentation module, and a heterogeneous attention module.
  arXiv  Detail & Related papers  (2022-01-03T02:32:06Z)
• Mesa: A Memory-saving Training Framework for Transformers [58.78933015299703]
  We present Mesa, a memory-saving training framework for Transformers. Mesa uses exact activations during forward pass while storing a low-precision version of activations to reduce memory consumption during training. Experiments on ImageNet, CIFAR-100 and ADE20K demonstrate that Mesa can reduce half of the memory footprints during training.
  arXiv  Detail & Related papers  (2021-11-22T11:23:01Z)
• Task Agnostic Metrics for Reservoir Computing [0.0]
  Physical reservoir computing is a computational paradigm that enables temporal pattern recognition in physical matter. The chosen dynamical system must have three desirable properties: non-linearity, complexity, and fading memory. We show that, in general, systems with lower damping reach higher values in all three performance metrics.
  arXiv  Detail & Related papers  (2021-08-03T13:58:11Z)
• Reservoir Stack Machines [77.12475691708838]
  Memory-augmented neural networks equip a recurrent neural network with an explicit memory to support tasks that require information storage. We introduce the reservoir stack machine, a model which can provably recognize all deterministic context-free languages. Our results show that the reservoir stack machine achieves zero error, even on test sequences longer than the training data.
  arXiv  Detail & Related papers  (2021-05-04T16:50:40Z)
• Temporal Memory Relation Network for Workflow Recognition from Surgical Video [53.20825496640025]
  We propose a novel end-to-end temporal memory relation network (TMNet) for relating long-range and multi-scale temporal patterns. We have extensively validated our approach on two benchmark surgical video datasets.
  arXiv  Detail & Related papers  (2021-03-30T13:20:26Z)
• Encoding-based Memory Modules for Recurrent Neural Networks [79.42778415729475]
  We study the memorization subtask from the point of view of the design and training of recurrent neural networks. We propose a new model, the Linear Memory Network, which features an encoding-based memorization component built with a linear autoencoder for sequences.
  arXiv  Detail & Related papers  (2020-01-31T11:14:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.