Map-based Experience Replay: A Memory-Efficient Solution to Catastrophic Forgetting in Reinforcement Learning

• URL: http://arxiv.org/abs/2305.02054v2
• Date: Mon, 28 Aug 2023 14:38:33 GMT
• Title: Map-based Experience Replay: A Memory-Efficient Solution to Catastrophic Forgetting in Reinforcement Learning
• Authors: Muhammad Burhan Hafez, Tilman Immisch, Tom Weber, Stefan Wermter
• Abstract summary: Deep Reinforcement Learning agents often suffer from catastrophic forgetting, forgetting previously found solutions in parts of the input space when training on new data. We introduce a novel cognitive-inspired replay memory approach based on the Grow-When-Required (GWR) self-organizing network. Our approach organizes stored transitions into a concise environment-model-like network of state-nodes and transition-edges, merging similar samples to reduce the memory size and increase pair-wise distance among samples.
• Score: 15.771773131031054
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deep Reinforcement Learning agents often suffer from catastrophic forgetting, forgetting previously found solutions in parts of the input space when training on new data. Replay Memories are a common solution to the problem, decorrelating and shuffling old and new training samples. They naively store state transitions as they come in, without regard for redundancy. We introduce a novel cognitive-inspired replay memory approach based on the Grow-When-Required (GWR) self-organizing network, which resembles a map-based mental model of the world. Our approach organizes stored transitions into a concise environment-model-like network of state-nodes and transition-edges, merging similar samples to reduce the memory size and increase pair-wise distance among samples, which increases the relevancy of each sample. Overall, our paper shows that map-based experience replay allows for significant memory reduction with only small performance decreases.

Related papers

• MsMemoryGAN: A Multi-scale Memory GAN for Palm-vein Adversarial Purification [40.80205521005344]
  We propose a novel defense model named MsMemoryGAN to filter the perturbations from adversarial samples before recognition. MsMemoryGAN learns to reconstruct the input by merely using fewer prototypical elements of the normal patterns recorded in the memory. Our approach removes a wide variety of adversarial perturbations, allowing vein classifiers to achieve the highest recognition accuracy.
  arXiv  Detail & Related papers  (2024-08-20T09:46:30Z)
• SHERL: Synthesizing High Accuracy and Efficient Memory for Resource-Limited Transfer Learning [63.93193829913252]
  We propose an innovative METL strategy called SHERL for resource-limited scenarios. In the early route, intermediate outputs are consolidated via an anti-redundancy operation. In the late route, utilizing minimal late pre-trained layers could alleviate the peak demand on memory overhead.
  arXiv  Detail & Related papers  (2024-07-10T10:22:35Z)
• TEAL: New Selection Strategy for Small Buffers in Experience Replay Class Incremental Learning [7.627299398469962]
  We introduce TEAL, a novel approach to populate the memory with exemplars. We show that TEAL improves the average accuracy of the SOTA method XDER as well as ER and ER-ACE on several image recognition benchmarks.
  arXiv  Detail & Related papers  (2024-06-30T12:09:08Z)
• Enhancing Consistency and Mitigating Bias: A Data Replay Approach for Incremental Learning [100.7407460674153]
  Deep learning systems are prone to catastrophic forgetting when learning from a sequence of tasks. To mitigate the problem, a line of methods propose to replay the data of experienced tasks when learning new tasks. However, it is not expected in practice considering the memory constraint or data privacy issue. As a replacement, data-free data replay methods are proposed by inverting samples from the classification model.
  arXiv  Detail & Related papers  (2024-01-12T12:51:12Z)
• Memory Replay with Data Compression for Continual Learning [80.95444077825852]
  We propose memory replay with data compression to reduce the storage cost of old training samples. We extensively validate this across several benchmarks of class-incremental learning and in a realistic scenario of object detection for autonomous driving.
  arXiv  Detail & Related papers  (2022-02-14T10:26:23Z)
• Saliency Guided Experience Packing for Replay in Continual Learning [6.417011237981518]
  We propose a new approach for experience replay, where we select the past experiences by looking at the saliency maps. While learning a new task, we replay these memory patches with appropriate zero-padding to remind the model about its past decisions.
  arXiv  Detail & Related papers  (2021-09-10T15:54:58Z)
• Reducing Representation Drift in Online Continual Learning [87.71558506591937]
  We study the online continual learning paradigm, where agents must learn from a changing distribution with constrained memory and compute. In this work we instead focus on the change in representations of previously observed data due to the introduction of previously unobserved class samples in the incoming data stream.
  arXiv  Detail & Related papers  (2021-04-11T15:19:30Z)
• Improving Computational Efficiency in Visual Reinforcement Learning via Stored Embeddings [89.63764845984076]
  We present Stored Embeddings for Efficient Reinforcement Learning (SEER) SEER is a simple modification of existing off-policy deep reinforcement learning methods. We show that SEER does not degrade the performance of RLizable agents while significantly saving computation and memory.
  arXiv  Detail & Related papers  (2021-03-04T08:14:10Z)
• The Effectiveness of Memory Replay in Large Scale Continual Learning [42.67483945072039]
  We study continual learning in the large scale setting where tasks in the input sequence are not limited to classification, and the outputs can be of high dimension. Existing methods usually replay only the input-output pairs. We propose to replay the activation of the intermediate layers in addition to the input-output pairs.
  arXiv  Detail & Related papers  (2020-10-06T01:23:12Z)
• Automatic Recall Machines: Internal Replay, Continual Learning and the Brain [104.38824285741248]
  Replay in neural networks involves training on sequential data with memorized samples, which counteracts forgetting of previous behavior caused by non-stationarity. We present a method where these auxiliary samples are generated on the fly, given only the model that is being trained for the assessed objective. Instead the implicit memory of learned samples within the assessed model itself is exploited.
  arXiv  Detail & Related papers  (2020-06-22T15:07:06Z)
• Fully reversible neural networks for large-scale surface and sub-surface characterization via remote sensing [4.383011485317949]
  Large spatial/frequency scale of hyperspectral and airborne magnetic and gravitational data causes memory issues when using convolutional neural networks for (sub-) surface characterization. We show examples from land-use change detection from hyperspectral time-lapse data, and regional aquifer mapping from airborne geophysical and geological data.
  arXiv  Detail & Related papers  (2020-03-16T23:54:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.