Deep Reinforcement Learning with Plasticity Injection

• URL: http://arxiv.org/abs/2305.15555v2
• Date: Tue, 3 Oct 2023 21:51:58 GMT
• Title: Deep Reinforcement Learning with Plasticity Injection
• Authors: Evgenii Nikishin, Junhyuk Oh, Georg Ostrovski, Clare Lyle, Razvan Pascanu, Will Dabney, Andr\'e Barreto
• Abstract summary: Evidence suggests that in deep reinforcement learning (RL) networks gradually lose their plasticity. plasticity injection increases the network plasticity without changing the number of parameters. plasticity injection attains stronger performance compared to alternative methods.
• Score: 37.19742321534183
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A growing body of evidence suggests that neural networks employed in deep reinforcement learning (RL) gradually lose their plasticity, the ability to learn from new data; however, the analysis and mitigation of this phenomenon is hampered by the complex relationship between plasticity, exploration, and performance in RL. This paper introduces plasticity injection, a minimalistic intervention that increases the network plasticity without changing the number of trainable parameters or biasing the predictions. The applications of this intervention are two-fold: first, as a diagnostic tool $\unicode{x2014}$ if injection increases the performance, we may conclude that an agent's network was losing its plasticity. This tool allows us to identify a subset of Atari environments where the lack of plasticity causes performance plateaus, motivating future studies on understanding and combating plasticity loss. Second, plasticity injection can be used to improve the computational efficiency of RL training if the agent has to re-learn from scratch due to exhausted plasticity or by growing the agent's network dynamically without compromising performance. The results on Atari show that plasticity injection attains stronger performance compared to alternative methods while being computationally efficient.

Related papers

• Plasticity Loss in Deep Reinforcement Learning: A Survey [15.525552360867367]
  plasticity is crucial for deep Reinforcement Learning (RL) agents. Once plasticity is lost, an agent's performance will plateau because it cannot improve its policy to account for changes in the data distribution. Loss of plasticity can be connected to many other issues plaguing deep RL, such as training instabilities, scaling failures, overestimation bias, and insufficient exploration.
  arXiv  Detail & Related papers  (2024-11-07T16:13:54Z)
• Disentangling the Causes of Plasticity Loss in Neural Networks [55.23250269007988]
  We show that loss of plasticity can be decomposed into multiple independent mechanisms. We show that a combination of layer normalization and weight decay is highly effective at maintaining plasticity in a variety of synthetic nonstationary learning tasks.
  arXiv  Detail & Related papers  (2024-02-29T00:02:33Z)
• Revisiting Plasticity in Visual Reinforcement Learning: Data, Modules and Training Stages [56.98243487769916]
  Plasticity, the ability of a neural network to evolve with new data, is crucial for high-performance and sample-efficient visual reinforcement learning. We propose Adaptive RR, which dynamically adjusts the replay ratio based on the critic's plasticity level.
  arXiv  Detail & Related papers  (2023-10-11T12:05:34Z)
• PLASTIC: Improving Input and Label Plasticity for Sample Efficient Reinforcement Learning [54.409634256153154]
  In Reinforcement Learning (RL), enhancing sample efficiency is crucial. In principle, off-policy RL algorithms can improve sample efficiency by allowing multiple updates per environment interaction. Our study investigates the underlying causes of this phenomenon by dividing plasticity into two aspects.
  arXiv  Detail & Related papers  (2023-06-19T06:14:51Z)
• Understanding plasticity in neural networks [41.79540750236036]
  Plasticity is the ability of a neural network to quickly change its predictions in response to new information. Deep neural networks are known to lose plasticity over the course of training even in relatively simple learning problems.
  arXiv  Detail & Related papers  (2023-03-02T18:47:51Z)
• An Adversarial Active Sampling-based Data Augmentation Framework for Manufacturable Chip Design [55.62660894625669]
  Lithography modeling is a crucial problem in chip design to ensure a chip design mask is manufacturable. Recent developments in machine learning have provided alternative solutions in replacing the time-consuming lithography simulations with deep neural networks. We propose a litho-aware data augmentation framework to resolve the dilemma of limited data and improve the machine learning model performance.
  arXiv  Detail & Related papers  (2022-10-27T20:53:39Z)
• Unveiling the role of plasticity rules in reservoir computing [0.0]
  Reservoir Computing (RC) is an appealing approach in Machine Learning. We analyze the role that plasticity rules play on the changes that lead to a better performance of RC.
  arXiv  Detail & Related papers  (2021-01-14T19:55:30Z)
• Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learning [97.28695683236981]
  More gradient updates decrease the expressivity of the current value network. We demonstrate this phenomenon on Atari and Gym benchmarks, in both offline and online RL settings.
  arXiv  Detail & Related papers  (2020-10-27T17:55:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.