Explainable Reinforcement Learning via a Causal World Model

• URL: http://arxiv.org/abs/2305.02749v5
• Date: Thu, 18 Jan 2024 09:57:32 GMT
• Title: Explainable Reinforcement Learning via a Causal World Model
• Authors: Zhongwei Yu, Jingqing Ruan, Dengpeng Xing
• Abstract summary: We learn a causal world model without prior knowledge of the causal structure of the environment. The model captures the influence of actions, allowing us to interpret the long-term effects of actions through causal chains. Our model remains accurate while improving explainability, making it applicable in model-based learning.
• Score: 5.4934134592053185
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Generating explanations for reinforcement learning (RL) is challenging as actions may produce long-term effects on the future. In this paper, we develop a novel framework for explainable RL by learning a causal world model without prior knowledge of the causal structure of the environment. The model captures the influence of actions, allowing us to interpret the long-term effects of actions through causal chains, which present how actions influence environmental variables and finally lead to rewards. Different from most explanatory models which suffer from low accuracy, our model remains accurate while improving explainability, making it applicable in model-based learning. As a result, we demonstrate that our causal model can serve as the bridge between explainability and learning.

Related papers

• Learning Nonlinear Causal Reductions to Explain Reinforcement Learning Policies [50.30741668990102]
  We take a causal perspective on explaining the behavior of reinforcement learning policies.<n>We learn a simplified high-level causal model that explains these relationships.<n>We prove that for a class of nonlinear causal models, there exists a unique solution.
  arXiv  Detail & Related papers  (2025-07-20T10:25:24Z)
• The Importance of Being Lazy: Scaling Limits of Continual Learning [60.97756735877614]
  We show that increasing model width is only beneficial when it reduces the amount of feature learning, yielding more laziness.<n>We study the intricate relationship between feature learning, task non-stationarity, and forgetting, finding that high feature learning is only beneficial with highly similar tasks.
  arXiv  Detail & Related papers  (2025-06-20T10:12:38Z)
• Causal-aware Large Language Models: Enhancing Decision-Making Through Learning, Adapting and Acting [25.351469582315975]
  Large language models (LLMs) have shown great potential in decision-making due to the vast amount of knowledge stored within the models.<n>We propose Causal-aware LLMs, which integrate the structural causal model (SCM) into the decision-making process to model, update, and utilize structured knowledge of the environment.
  arXiv  Detail & Related papers  (2025-05-30T15:30:44Z)
• Causal Distillation: Transferring Structured Explanations from Large to Compact Language Models [0.0]
  Large proprietary language models exhibit strong causal reasoning abilities that smaller open-source models struggle to replicate.<n>We introduce a novel framework for distilling causal explanations that transfers causal reasoning skills from a powerful teacher model to a compact open-source model.<n>The key idea is to train the smaller model to develop causal reasoning abilities by generating structured cause-and-effect explanations consistent with those of the teacher model.
  arXiv  Detail & Related papers  (2025-05-26T04:50:42Z)
• Towards Empowerment Gain through Causal Structure Learning in Model-Based RL [35.933469787075]
  We propose a novel framework, Empowerment through Causal Learning (ECL), to improve learning efficiency and controllability. ECL operates by first training a causal dynamics model of the environment based on collected data. We then maximize empowerment under the causal structure for exploration, simultaneously using data gathered through exploration to update causal dynamics model to be more controllable.
  arXiv  Detail & Related papers  (2025-02-14T10:59:09Z)
• Failure Modes of LLMs for Causal Reasoning on Narratives [51.19592551510628]
  We investigate the interaction between world knowledge and logical reasoning.<n>We find that state-of-the-art large language models (LLMs) often rely on superficial generalizations.<n>We show that simple reformulations of the task can elicit more robust reasoning behavior.
  arXiv  Detail & Related papers  (2024-10-31T12:48:58Z)
• DAG-aware Transformer for Causal Effect Estimation [0.8192907805418583]
  Causal inference is a critical task across fields such as healthcare, economics, and the social sciences. In this paper, we present a novel transformer-based method for causal inference that overcomes these challenges. The core innovation of our model lies in its integration of causal Directed Acyclic Graphs (DAGs) directly into the attention mechanism.
  arXiv  Detail & Related papers  (2024-10-13T23:17:58Z)
• On the Modeling Capabilities of Large Language Models for Sequential Decision Making [52.128546842746246]
  Large pretrained models are showing increasingly better performance in reasoning and planning tasks. We evaluate their ability to produce decision-making policies, either directly, by generating actions, or indirectly. In environments with unfamiliar dynamics, we explore how fine-tuning LLMs with synthetic data can significantly improve their reward modeling capabilities.
  arXiv  Detail & Related papers  (2024-10-08T03:12:57Z)
• Causal Graph Neural Networks for Wildfire Danger Prediction [25.12733727343395]
  Wildfire forecasting is notoriously hard due to the complex interplay of different factors such as weather conditions, vegetation types and human activities. Deep learning models show promise in dealing with this complexity by learning directly from data. We argue that we need models that are right for the right reasons; that is, the implicit rules learned should be grounded by the underlying processes driving wildfires.
  arXiv  Detail & Related papers  (2024-03-13T10:58:55Z)
• Identifiable Latent Polynomial Causal Models Through the Lens of Change [82.14087963690561]
  Causal representation learning aims to unveil latent high-level causal representations from observed low-level data. One of its primary tasks is to provide reliable assurance of identifying these latent causal models, known as identifiability.
  arXiv  Detail & Related papers  (2023-10-24T07:46:10Z)
• Interpretable Imitation Learning with Dynamic Causal Relations [65.18456572421702]
  We propose to expose captured knowledge in the form of a directed acyclic causal graph. We also design this causal discovery process to be state-dependent, enabling it to model the dynamics in latent causal graphs. The proposed framework is composed of three parts: a dynamic causal discovery module, a causality encoding module, and a prediction module, and is trained in an end-to-end manner.
  arXiv  Detail & Related papers  (2023-09-30T20:59:42Z)
• Minimal Value-Equivalent Partial Models for Scalable and Robust Planning in Lifelong Reinforcement Learning [56.50123642237106]
  Common practice in model-based reinforcement learning is to learn models that model every aspect of the agent's environment. We argue that such models are not particularly well-suited for performing scalable and robust planning in lifelong reinforcement learning scenarios. We propose new kinds of models that only model the relevant aspects of the environment, which we call "minimal value-minimal partial models"
  arXiv  Detail & Related papers  (2023-01-24T16:40:01Z)
• Towards Interpretable Deep Reinforcement Learning Models via Inverse Reinforcement Learning [27.841725567976315]
  We propose a novel framework utilizing Adversarial Inverse Reinforcement Learning. This framework provides global explanations for decisions made by a Reinforcement Learning model. We capture intuitive tendencies that the model follows by summarizing the model's decision-making process.
  arXiv  Detail & Related papers  (2022-03-30T17:01:59Z)
• Causality in Neural Networks -- An Extended Abstract [0.0]
  Causal reasoning is the main learning and explanation tool used by humans. Introducing the ideas of causality to machine learning helps in providing better learning and explainable models.
  arXiv  Detail & Related papers  (2021-06-03T09:52:36Z)
• Remembering for the Right Reasons: Explanations Reduce Catastrophic Forgetting [100.75479161884935]
  We propose a novel training paradigm called Remembering for the Right Reasons (RRR) RRR stores visual model explanations for each example in the buffer and ensures the model has "the right reasons" for its predictions. We demonstrate how RRR can be easily added to any memory or regularization-based approach and results in reduced forgetting.
  arXiv  Detail & Related papers  (2020-10-04T10:05:27Z)
• Towards Interpretable Reasoning over Paragraph Effects in Situation [126.65672196760345]
  We focus on the task of reasoning over paragraph effects in situation, which requires a model to understand the cause and effect. We propose a sequential approach for this task which explicitly models each step of the reasoning process with neural network modules. In particular, five reasoning modules are designed and learned in an end-to-end manner, which leads to a more interpretable model.
  arXiv  Detail & Related papers  (2020-10-03T04:03:52Z)
• Distal Explanations for Model-free Explainable Reinforcement Learning [19.250329276538352]
  We introduce and evaluate a distal explanation model for model-free reinforcement learning agents. Our starting point is the observation that causal models can generate opportunity chains that take the form of A enables B and B causes C'
  arXiv  Detail & Related papers  (2020-01-28T11:57:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.