Revisiting Plasticity in Visual Reinforcement Learning: Data, Modules and Training Stages

• URL: http://arxiv.org/abs/2310.07418v3
• Date: Sun, 19 May 2024 19:04:31 GMT
• Title: Revisiting Plasticity in Visual Reinforcement Learning: Data, Modules and Training Stages
• Authors: Guozheng Ma, Lu Li, Sen Zhang, Zixuan Liu, Zhen Wang, Yixin Chen, Li Shen, Xueqian Wang, Dacheng Tao,
• Abstract summary: Plasticity, the ability of a neural network to evolve with new data, is crucial for high-performance and sample-efficient visual reinforcement learning. We propose Adaptive RR, which dynamically adjusts the replay ratio based on the critic's plasticity level.
• Score: 56.98243487769916
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Plasticity, the ability of a neural network to evolve with new data, is crucial for high-performance and sample-efficient visual reinforcement learning (VRL). Although methods like resetting and regularization can potentially mitigate plasticity loss, the influences of various components within the VRL framework on the agent's plasticity are still poorly understood. In this work, we conduct a systematic empirical exploration focusing on three primary underexplored facets and derive the following insightful conclusions: (1) data augmentation is essential in maintaining plasticity; (2) the critic's plasticity loss serves as the principal bottleneck impeding efficient training; and (3) without timely intervention to recover critic's plasticity in the early stages, its loss becomes catastrophic. These insights suggest a novel strategy to address the high replay ratio (RR) dilemma, where exacerbated plasticity loss hinders the potential improvements of sample efficiency brought by increased reuse frequency. Rather than setting a static RR for the entire training process, we propose Adaptive RR, which dynamically adjusts the RR based on the critic's plasticity level. Extensive evaluations indicate that Adaptive RR not only avoids catastrophic plasticity loss in the early stages but also benefits from more frequent reuse in later phases, resulting in superior sample efficiency.

Related papers

• Dynamic Loss-Based Sample Reweighting for Improved Large Language Model Pretraining [55.262510814326035]
  Existing reweighting strategies primarily focus on group-level data importance. We introduce novel algorithms for dynamic, instance-level data reweighting. Our framework allows us to devise reweighting strategies deprioritizing redundant or uninformative data.
  arXiv  Detail & Related papers  (2025-02-10T17:57:15Z)
• Feasible Learning [78.6167929413604]
  We introduce Feasible Learning (FL), a sample-centric learning paradigm where models are trained by solving a feasibility problem that bounds the loss for each training sample. Our empirical analysis, spanning image classification, age regression, and preference optimization in large language models, demonstrates that models trained via FL can learn from data while displaying improved tail behavior compared to ERM, with only a marginal impact on average performance.
  arXiv  Detail & Related papers  (2025-01-24T20:39:38Z)
• Plasticity Loss in Deep Reinforcement Learning: A Survey [15.525552360867367]
  plasticity is crucial for deep Reinforcement Learning (RL) agents. Once plasticity is lost, an agent's performance will plateau because it cannot improve its policy to account for changes in the data distribution. Loss of plasticity can be connected to many other issues plaguing deep RL, such as training instabilities, scaling failures, overestimation bias, and insufficient exploration.
  arXiv  Detail & Related papers  (2024-11-07T16:13:54Z)
• Disentangling the Causes of Plasticity Loss in Neural Networks [55.23250269007988]
  We show that loss of plasticity can be decomposed into multiple independent mechanisms. We show that a combination of layer normalization and weight decay is highly effective at maintaining plasticity in a variety of synthetic nonstationary learning tasks.
  arXiv  Detail & Related papers  (2024-02-29T00:02:33Z)
• Retrosynthesis prediction enhanced by in-silico reaction data augmentation [66.5643280109899]
  We present RetroWISE, a framework that employs a base model inferred from real paired data to perform in-silico reaction generation and augmentation. On three benchmark datasets, RetroWISE achieves the best overall performance against state-of-the-art models.
  arXiv  Detail & Related papers  (2024-01-31T07:40:37Z)
• PLASTIC: Improving Input and Label Plasticity for Sample Efficient Reinforcement Learning [54.409634256153154]
  In Reinforcement Learning (RL), enhancing sample efficiency is crucial. In principle, off-policy RL algorithms can improve sample efficiency by allowing multiple updates per environment interaction. Our study investigates the underlying causes of this phenomenon by dividing plasticity into two aspects.
  arXiv  Detail & Related papers  (2023-06-19T06:14:51Z)
• Deep Reinforcement Learning with Plasticity Injection [37.19742321534183]
  Evidence suggests that in deep reinforcement learning (RL) networks gradually lose their plasticity. plasticity injection increases the network plasticity without changing the number of parameters. plasticity injection attains stronger performance compared to alternative methods.
  arXiv  Detail & Related papers  (2023-05-24T20:41:35Z)
• Understanding and Preventing Capacity Loss in Reinforcement Learning [28.52122927103544]
  We identify a mechanism by which non-stationary prediction targets can prevent learning progress in deep RL agents. Capacity loss occurs in a range of RL agents and environments, and is particularly damaging to performance in sparse-reward tasks.
  arXiv  Detail & Related papers  (2022-04-20T15:55:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.