Improvements of Dark Experience Replay and Reservoir Sampling towards Better Balance between Consolidation and Plasticity

• URL: http://arxiv.org/abs/2504.20932v1
• Date: Tue, 29 Apr 2025 16:50:05 GMT
• Title: Improvements of Dark Experience Replay and Reservoir Sampling towards Better Balance between Consolidation and Plasticity
• Authors: Taisuke Kobayashi,
• Abstract summary: Continual learning is one of the most essential abilities for autonomous agents.<n>The ability to retain past outputs inhibits learning if the past outputs are incorrect due to distribution shift or other effects.<n>This paper proposes improvement strategies to each of DER and RS.
• Score: 6.20048328543366
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Continual learning is the one of the most essential abilities for autonomous agents, which can incrementally learn daily-life skills. For this ultimate goal, a simple but powerful method, dark experience replay (DER), has been proposed recently. DER mitigates catastrophic forgetting, in which the skills acquired in the past are unintentionally forgotten, by stochastically storing the streaming data in a reservoir sampling (RS) buffer and by relearning them or retaining the past outputs for them. However, since DER considers multiple objectives, it will not function properly without appropriate weighting of them. In addition, the ability to retain past outputs inhibits learning if the past outputs are incorrect due to distribution shift or other effects. This is due to a tradeoff between memory consolidation and plasticity. The tradeoff is hidden even in the RS buffer, which gradually stops storing new data for new skills in it as data is continuously passed to it. To alleviate the tradeoff and achieve better balance, this paper proposes improvement strategies to each of DER and RS. Specifically, DER is improved with automatic adaptation of weights, block of replaying erroneous data, and correction of past outputs. RS is also improved with generalization of acceptance probability, stratification of plural buffers, and intentional omission of unnecessary data. These improvements are verified through multiple benchmarks including regression, classification, and reinforcement learning problems. As a result, the proposed methods achieve steady improvements in learning performance by balancing the memory consolidation and plasticity.

Related papers

• May the Forgetting Be with You: Alternate Replay for Learning with Noisy Labels [16.262555459431155]
  We introduce Alternate Experience Replay (AER), which takes advantage of forgetting to maintain a clear distinction between clean, complex, and noisy samples in the memory buffer. We demonstrate the effectiveness of our approach in terms of both accuracy and purity of the obtained buffer, resulting in a remarkable average gain of 4.71% points in accuracy with respect to existing loss-based purification strategies.
  arXiv  Detail & Related papers  (2024-08-26T14:09:40Z)
• Improving Data-aware and Parameter-aware Robustness for Continual Learning [3.480626767752489]
  This paper analyzes that this insufficiency arises from the ineffective handling of outliers. We propose a Robust Continual Learning (RCL) method to address this issue. The proposed method effectively maintains robustness and achieves new state-of-the-art (SOTA) results.
  arXiv  Detail & Related papers  (2024-05-27T11:21:26Z)
• Enhancing Consistency and Mitigating Bias: A Data Replay Approach for Incremental Learning [93.90047628101155]
  Deep learning systems are prone to catastrophic forgetting when learning from a sequence of tasks.<n>To address this, some methods propose replaying data from previous tasks during new task learning.<n>However, it is not expected in practice due to memory constraints and data privacy issues.
  arXiv  Detail & Related papers  (2024-01-12T12:51:12Z)
• Fast Machine Unlearning Without Retraining Through Selective Synaptic Dampening [51.34904967046097]
  Selective Synaptic Dampening (SSD) is a fast, performant, and does not require long-term storage of the training data. We present a novel two-step, post hoc, retrain-free approach to machine unlearning which is fast, performant, and does not require long-term storage of the training data.
  arXiv  Detail & Related papers  (2023-08-15T11:30:45Z)
• Balanced Destruction-Reconstruction Dynamics for Memory-replay Class Incremental Learning [27.117753965919025]
  Class incremental learning (CIL) aims to incrementally update a trained model with the new classes of samples. Memory-replay CIL consolidates old knowledge by replaying a small number of old classes of samples saved in the memory. Our theoretical analysis shows that the destruction of old knowledge can be effectively alleviated by balancing the contribution of samples from the current phase and those saved in the memory.
  arXiv  Detail & Related papers  (2023-08-03T11:33:50Z)
• Improving information retention in large scale online continual learning [99.73847522194549]
  Online continual learning aims to adapt efficiently to new data while retaining existing knowledge. Recent work suggests that information retention remains a problem in large scale OCL even when the replay buffer is unlimited. We propose using a moving average family of methods to improve optimization for non-stationary objectives.
  arXiv  Detail & Related papers  (2022-10-12T16:59:43Z)
• Queried Unlabeled Data Improves and Robustifies Class-Incremental Learning [133.39254981496146]
  Class-incremental learning (CIL) suffers from the notorious dilemma between learning newly added classes and preserving previously learned class knowledge. We propose to leverage "free" external unlabeled data querying in continual learning. We show queried unlabeled data can continue to benefit, and seamlessly extend CIL-QUD into its robustified versions.
  arXiv  Detail & Related papers  (2022-06-15T22:53:23Z)
• Learning Bayesian Sparse Networks with Full Experience Replay for Continual Learning [54.7584721943286]
  Continual Learning (CL) methods aim to enable machine learning models to learn new tasks without catastrophic forgetting of those that have been previously mastered. Existing CL approaches often keep a buffer of previously-seen samples, perform knowledge distillation, or use regularization techniques towards this goal. We propose to only activate and select sparse neurons for learning current and past tasks at any stage.
  arXiv  Detail & Related papers  (2022-02-21T13:25:03Z)
• Improving Computational Efficiency in Visual Reinforcement Learning via Stored Embeddings [89.63764845984076]
  We present Stored Embeddings for Efficient Reinforcement Learning (SEER) SEER is a simple modification of existing off-policy deep reinforcement learning methods. We show that SEER does not degrade the performance of RLizable agents while significantly saving computation and memory.
  arXiv  Detail & Related papers  (2021-03-04T08:14:10Z)
• Adaptive and Multiple Time-scale Eligibility Traces for Online Deep Reinforcement Learning [8.071506311915396]
  The eligibility traces method is well known as an online learning technique for improving sample efficiency. The dependency between parameters of deep neural networks would destroy the eligibility traces, which is why they are not integrated with DRL. This study proposes a new eligibility traces method that can be used even in DRL while maintaining high sample efficiency.
  arXiv  Detail & Related papers  (2020-08-23T13:35:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.