CIER: A Novel Experience Replay Approach with Causal Inference in Deep Reinforcement Learning

• URL: http://arxiv.org/abs/2405.08380v1
• Date: Tue, 14 May 2024 07:23:10 GMT
• Title: CIER: A Novel Experience Replay Approach with Causal Inference in Deep Reinforcement Learning
• Authors: Jingwen Wang, Dehui Du, Yida Li, Yiyang Li, Yikang Chen,
• Abstract summary: We propose a novel approach to segment time series into meaningful subsequences and represent the time series based on these subsequences. The subsequences are employed for causal inference to identify fundamental causal factors that significantly impact training outcomes. Several experiments demonstrate the feasibility of our approach in common environments, confirming its ability to enhance the effectiveness of DRL training and impart a certain level of explainability to the training process.
• Score: 11.13226491866178
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In the training process of Deep Reinforcement Learning (DRL), agents require repetitive interactions with the environment. With an increase in training volume and model complexity, it is still a challenging problem to enhance data utilization and explainability of DRL training. This paper addresses these challenges by focusing on the temporal correlations within the time dimension of time series. We propose a novel approach to segment multivariate time series into meaningful subsequences and represent the time series based on these subsequences. Furthermore, the subsequences are employed for causal inference to identify fundamental causal factors that significantly impact training outcomes. We design a module to provide feedback on the causality during DRL training. Several experiments demonstrate the feasibility of our approach in common environments, confirming its ability to enhance the effectiveness of DRL training and impart a certain level of explainability to the training process. Additionally, we extended our approach with priority experience replay algorithm, and experimental results demonstrate the continued effectiveness of our approach.

Related papers

• Symmetric Reinforcement Learning Loss for Robust Learning on Diverse Tasks and Model Scales [13.818149654692863]
  Reinforcement learning (RL) training is inherently unstable due to factors such as moving targets and high gradient variance. In this work, we improve the stability of RL training by adapting the reverse cross entropy (RCE) from supervised learning for noisy data to define a symmetric RL loss.
  arXiv  Detail & Related papers  (2024-05-27T19:28:33Z)
• Dissecting Deep RL with High Update Ratios: Combatting Value Divergence [21.282292112642747]
  We show that deep reinforcement learning algorithms can retain their ability to learn without resetting network parameters. We employ a simple unit-ball normalization that enables learning under large update ratios.
  arXiv  Detail & Related papers  (2024-03-09T19:56:40Z)
• Replay across Experiments: A Natural Extension of Off-Policy RL [18.545939667810565]
  We present an effective yet simple framework to extend the use of replays across multiple experiments. At its core, Replay Across Experiments (RaE) involves reusing experience from previous experiments to improve exploration and bootstrap learning. We empirically show benefits across a number of RL algorithms and challenging control domains spanning both locomotion and manipulation.
  arXiv  Detail & Related papers  (2023-11-27T15:57:11Z)
• Repetition In Repetition Out: Towards Understanding Neural Text Degeneration from the Data Perspective [91.14291142262262]
  This work presents a straightforward and fundamental explanation from the data perspective. Our preliminary investigation reveals a strong correlation between the degeneration issue and the presence of repetitions in training data. Our experiments reveal that penalizing the repetitions in training data remains critical even when considering larger model sizes and instruction tuning.
  arXiv  Detail & Related papers  (2023-10-16T09:35:42Z)
• Learning Dynamics and Generalization in Reinforcement Learning [59.530058000689884]
  We show theoretically that temporal difference learning encourages agents to fit non-smooth components of the value function early in training. We show that neural networks trained using temporal difference algorithms on dense reward tasks exhibit weaker generalization between states than randomly networks and gradient networks trained with policy methods.
  arXiv  Detail & Related papers  (2022-06-05T08:49:16Z)
• Stratified Experience Replay: Correcting Multiplicity Bias in Off-Policy Reinforcement Learning [17.3794999533024]
  We show that deep RL appears to struggle in the presence of extraneous data. Recent works have shown that the performance of Deep Q-Network (DQN) degrades when its replay memory becomes too large. We re-examine the motivation for sampling uniformly over a replay memory, and find that it may be flawed when using function approximation.
  arXiv  Detail & Related papers  (2021-02-22T19:29:18Z)
• Causal Inference Q-Network: Toward Resilient Reinforcement Learning [57.96312207429202]
  We consider a resilient DRL framework with observational interferences. Under this framework, we propose a causal inference based DRL algorithm called causal inference Q-network (CIQ) Our experimental results show that the proposed CIQ method could achieve higher performance and more resilience against observational interferences.
  arXiv  Detail & Related papers  (2021-02-18T23:50:20Z)
• Dynamics Generalization via Information Bottleneck in Deep Reinforcement Learning [90.93035276307239]
  We propose an information theoretic regularization objective and an annealing-based optimization method to achieve better generalization ability in RL agents. We demonstrate the extreme generalization benefits of our approach in different domains ranging from maze navigation to robotic tasks. This work provides a principled way to improve generalization in RL by gradually removing information that is redundant for task-solving.
  arXiv  Detail & Related papers  (2020-08-03T02:24:20Z)
• Deep Reinforcement Learning using Cyclical Learning Rates [62.19441737665902]
  One of the most influential parameters in optimization procedures based on gradient descent (SGD) is the learning rate. We investigate cyclical learning and propose a method for defining a general cyclical learning rate for various DRL problems. Our experiments show that, utilizing cyclical learning achieves similar or even better results than highly tuned fixed learning rates.
  arXiv  Detail & Related papers  (2020-07-31T10:06:02Z)
• Transient Non-Stationarity and Generalisation in Deep Reinforcement Learning [67.34810824996887]
  Non-stationarity can arise in Reinforcement Learning (RL) even in stationary environments. We propose Iterated Relearning (ITER) to improve generalisation of deep RL agents.
  arXiv  Detail & Related papers  (2020-06-10T13:26:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.