Causal Reinforcement Learning: A Survey

• URL: http://arxiv.org/abs/2307.01452v2
• Date: Tue, 21 Nov 2023 03:43:15 GMT
• Title: Causal Reinforcement Learning: A Survey
• Authors: Zhihong Deng, Jing Jiang, Guodong Long, Chengqi Zhang
• Abstract summary: Reinforcement learning is an essential paradigm for solving sequential decision problems under uncertainty. One of the main obstacles is that reinforcement learning agents lack a fundamental understanding of the world. Causality offers a notable advantage as it can formalize knowledge in a systematic manner.
• Score: 57.368108154871
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement learning is an essential paradigm for solving sequential decision problems under uncertainty. Despite many remarkable achievements in recent decades, applying reinforcement learning methods in the real world remains challenging. One of the main obstacles is that reinforcement learning agents lack a fundamental understanding of the world and must therefore learn from scratch through numerous trial-and-error interactions. They may also face challenges in providing explanations for their decisions and generalizing the acquired knowledge. Causality, however, offers a notable advantage as it can formalize knowledge in a systematic manner and leverage invariance for effective knowledge transfer. This has led to the emergence of causal reinforcement learning, a subfield of reinforcement learning that seeks to enhance existing algorithms by incorporating causal relationships into the learning process. In this survey, we comprehensively review the literature on causal reinforcement learning. We first introduce the basic concepts of causality and reinforcement learning, and then explain how causality can address core challenges in non-causal reinforcement learning. We categorize and systematically review existing causal reinforcement learning approaches based on their target problems and methodologies. Finally, we outline open issues and future directions in this emerging field.

Related papers

• Reinforcement Learning with Knowledge Representation and Reasoning: A Brief Survey [24.81327556378729]
  Reinforcement Learning has achieved tremendous development in recent years. Still faces significant obstacles in addressing complex real-life problems. Recently, there has been a rapidly growing interest in the use of Knowledge Representation and Reasoning.
  arXiv  Detail & Related papers  (2023-04-24T13:35:11Z)
• Learning by Applying: A General Framework for Mathematical Reasoning via Enhancing Explicit Knowledge Learning [47.96987739801807]
  We propose a framework to enhance existing models (backbones) in a principled way by explicit knowledge learning. In LeAp, we perform knowledge learning in a novel problem-knowledge-expression paradigm. We show that LeAp improves all backbones' performances, learns accurate knowledge, and achieves a more interpretable reasoning process.
  arXiv  Detail & Related papers  (2023-02-11T15:15:41Z)
• Deep Causal Learning: Representation, Discovery and Inference [2.696435860368848]
  Causal learning reveals the essential relationships that underpin phenomena and delineates the mechanisms by which the world evolves. Traditional causal learning methods face numerous challenges and limitations, including high-dimensional variables, unstructured variables, optimization problems, unobserved confounders, selection biases, and estimation inaccuracies. Deep causal learning, which leverages deep neural networks, offers innovative insights and solutions for addressing these challenges.
  arXiv  Detail & Related papers  (2022-11-07T09:00:33Z)
• Anti-Retroactive Interference for Lifelong Learning [65.50683752919089]
  We design a paradigm for lifelong learning based on meta-learning and associative mechanism of the brain. It tackles the problem from two aspects: extracting knowledge and memorizing knowledge. It is theoretically analyzed that the proposed learning paradigm can make the models of different tasks converge to the same optimum.
  arXiv  Detail & Related papers  (2022-08-27T09:27:36Z)
• Rethinking Learning Dynamics in RL using Adversarial Networks [79.56118674435844]
  We present a learning mechanism for reinforcement learning of closely related skills parameterized via a skill embedding space. The main contribution of our work is to formulate an adversarial training regime for reinforcement learning with the help of entropy-regularized policy gradient formulation.
  arXiv  Detail & Related papers  (2022-01-27T19:51:09Z)
• Incremental Knowledge Based Question Answering [52.041815783025186]
  We propose a new incremental KBQA learning framework that can progressively expand learning capacity as humans do. Specifically, it comprises a margin-distilled loss and a collaborative selection method, to overcome the catastrophic forgetting problem. The comprehensive experiments demonstrate its effectiveness and efficiency when working with the evolving knowledge base.
  arXiv  Detail & Related papers  (2021-01-18T09:03:38Z)
• Transfer Learning in Deep Reinforcement Learning: A Survey [64.36174156782333]
  Reinforcement learning is a learning paradigm for solving sequential decision-making problems. Recent years have witnessed remarkable progress in reinforcement learning upon the fast development of deep neural networks. transfer learning has arisen to tackle various challenges faced by reinforcement learning.
  arXiv  Detail & Related papers  (2020-09-16T18:38:54Z)
• Importance Weighted Policy Learning and Adaptation [89.46467771037054]
  We study a complementary approach which is conceptually simple, general, modular and built on top of recent improvements in off-policy learning. The framework is inspired by ideas from the probabilistic inference literature and combines robust off-policy learning with a behavior prior. Our approach achieves competitive adaptation performance on hold-out tasks compared to meta reinforcement learning baselines and can scale to complex sparse-reward scenarios.
  arXiv  Detail & Related papers  (2020-09-10T14:16:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.