Solving Continuous Control with Episodic Memory

• URL: http://arxiv.org/abs/2106.08832v1
• Date: Wed, 16 Jun 2021 14:51:39 GMT
• Title: Solving Continuous Control with Episodic Memory
• Authors: Igor Kuznetsov, Andrey Filchenkov
• Abstract summary: Episodic memory lets reinforcement learning algorithms remember and exploit promising experience from the past to improve agent performance. Our study aims to answer the question: can episodic memory be used to improve agent's performance in continuous control?
• Score: 1.9493449206135294
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Episodic memory lets reinforcement learning algorithms remember and exploit promising experience from the past to improve agent performance. Previous works on memory mechanisms show benefits of using episodic-based data structures for discrete action problems in terms of sample-efficiency. The application of episodic memory for continuous control with a large action space is not trivial. Our study aims to answer the question: can episodic memory be used to improve agent's performance in continuous control? Our proposed algorithm combines episodic memory with Actor-Critic architecture by modifying critic's objective. We further improve performance by introducing episodic-based replay buffer prioritization. We evaluate our algorithm on OpenAI gym domains and show greater sample-efficiency compared with the state-of-the art model-free off-policy algorithms.

Related papers

• Addressing Issues with Working Memory in Video Object Segmentation [37.755852787082254]
  Video object segmentation (VOS) models compare incoming unannotated images to a history of image-mask relations. Current state of the art models perform very well on clean video data. Their reliance on a working memory of previous frames leaves room for error. A simple algorithmic change is proposed that can be applied to any existing working memory-based VOS model.
  arXiv  Detail & Related papers  (2024-10-29T18:34:41Z)
• Stable Hadamard Memory: Revitalizing Memory-Augmented Agents for Reinforcement Learning [64.93848182403116]
  Current deep-learning memory models struggle in reinforcement learning environments that are partially observable and long-term. We introduce the Stable Hadamard Memory, a novel memory model for reinforcement learning agents. Our approach significantly outperforms state-of-the-art memory-based methods on challenging partially observable benchmarks.
  arXiv  Detail & Related papers  (2024-10-14T03:50:17Z)
• Spatially-Aware Transformer for Embodied Agents [20.498778205143477]
  This paper explores the use of Spatially-Aware Transformer models that incorporate spatial information. We demonstrate that memory utilization efficiency can be improved, leading to enhanced accuracy in various place-centric downstream tasks. We also propose the Adaptive Memory Allocator, a memory management method based on reinforcement learning.
  arXiv  Detail & Related papers  (2024-02-23T07:46:30Z)
• Constant Memory Attention Block [74.38724530521277]
  Constant Memory Attention Block (CMAB) is a novel general-purpose attention block that computes its output in constant memory and performs updates in constant computation. We show our proposed methods achieve results competitive with state-of-the-art while being significantly more memory efficient.
  arXiv  Detail & Related papers  (2023-06-21T22:41:58Z)
• Think Before You Act: Decision Transformers with Working Memory [44.18926449252084]
  Decision Transformer-based decision-making agents have shown the ability to generalize across multiple tasks. We argue that this inefficiency stems from the forgetting phenomenon, in which a model memorizes its behaviors in parameters throughout training. We propose a working memory module to store, blend, and retrieve information for different downstream tasks.
  arXiv  Detail & Related papers  (2023-05-24T01:20:22Z)
• Sequential memory improves sample and memory efficiency in Episodic Control [0.0]
  State of the art deep reinforcement learning algorithms are sample inefficient due to the large number of episodes they require to achieve performance. ERL algorithms, inspired by the mammalian hippocampus, typically use extended memory systems to bootstrap learning from past events to overcome this sample-inefficiency problem. Here, we demonstrate that including a bias in the acquired memory content derived from the order of episodic sampling improves both the sample and memory efficiency of an episodic control algorithm.
  arXiv  Detail & Related papers  (2021-12-29T18:42:15Z)
• ABC: Attention with Bounded-memory Control [67.40631793251997]
  We show that bounded-memory control (ABC) can be subsumed into one abstraction, attention with bounded-memory control (ABC) ABC reveals new, unexplored possibilities. First, it connects several efficient attention variants that would otherwise seem apart. Last, we present a new instance of ABC, which draws inspiration from existing ABC approaches, but replaces their memory-organizing functions with a learned, contextualized one.
  arXiv  Detail & Related papers  (2021-10-06T03:53:25Z)
• Memory-Based Semantic Parsing [79.48882899104997]
  We present a memory-based model for context-dependent semantic parsing. We learn a context memory controller that manages the memory by maintaining the cumulative meaning of sequential user utterances.
  arXiv  Detail & Related papers  (2021-09-07T16:15:13Z)
• Generalizable Episodic Memory for Deep Reinforcement Learning [22.375796383623566]
  We propose Generalizable Episodic Memory (GEM), which effectively organizes the state-action values of episodic memory in a generalizable manner. GEM supports implicit planning on memorized trajectories. Empirical evaluation shows that our method significantly outperforms existing trajectory-based methods on various MuJoCo continuous control tasks.
  arXiv  Detail & Related papers  (2021-03-11T05:31:21Z)
• Improving Computational Efficiency in Visual Reinforcement Learning via Stored Embeddings [89.63764845984076]
  We present Stored Embeddings for Efficient Reinforcement Learning (SEER) SEER is a simple modification of existing off-policy deep reinforcement learning methods. We show that SEER does not degrade the performance of RLizable agents while significantly saving computation and memory.
  arXiv  Detail & Related papers  (2021-03-04T08:14:10Z)
• Memformer: A Memory-Augmented Transformer for Sequence Modeling [55.780849185884996]
  We present Memformer, an efficient neural network for sequence modeling. Our model achieves linear time complexity and constant memory space complexity when processing long sequences.
  arXiv  Detail & Related papers  (2020-10-14T09:03:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.