Learning by Doing: An Online Causal Reinforcement Learning Framework with Causal-Aware Policy

• URL: http://arxiv.org/abs/2402.04869v2
• Date: Thu, 24 Apr 2025 07:58:03 GMT
• Title: Learning by Doing: An Online Causal Reinforcement Learning Framework with Causal-Aware Policy
• Authors: Ruichu Cai, Siyang Huang, Jie Qiao, Wei Chen, Yan Zeng, Keli Zhang, Fuchun Sun, Yang Yu, Zhifeng Hao,
• Abstract summary: We consider explicitly modeling the generation process of states with the graphical causal model.<n>We formulate the causal structure updating into the RL interaction process with active intervention learning of the environment.
• Score: 38.86867078596718
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As a key component to intuitive cognition and reasoning solutions in human intelligence, causal knowledge provides great potential for reinforcement learning (RL) agents' interpretability towards decision-making by helping reduce the searching space. However, there is still a considerable gap in discovering and incorporating causality into RL, which hinders the rapid development of causal RL. In this paper, we consider explicitly modeling the generation process of states with the causal graphical model, based on which we augment the policy. We formulate the causal structure updating into the RL interaction process with active intervention learning of the environment. To optimize the derived objective, we propose a framework with theoretical performance guarantees that alternates between two steps: using interventions for causal structure learning during exploration and using the learned causal structure for policy guidance during exploitation. Due to the lack of public benchmarks that allow direct intervention in the state space, we design the root cause localization task in our simulated fault alarm environment and then empirically show the effectiveness and robustness of the proposed method against state-of-the-art baselines. Theoretical analysis shows that our performance improvement attributes to the virtuous cycle of causal-guided policy learning and causal structure learning, which aligns with our experimental results. Codes are available at https://github.com/DMIRLAB-Group/FaultAlarm_RL.

Related papers

• Towards Empowerment Gain through Causal Structure Learning in Model-Based RL [35.933469787075]
  We propose a novel framework, Empowerment through Causal Learning (ECL), to improve learning efficiency and controllability. ECL operates by first training a causal dynamics model of the environment based on collected data. We then maximize empowerment under the causal structure for exploration, simultaneously using data gathered through exploration to update causal dynamics model to be more controllable.
  arXiv  Detail & Related papers  (2025-02-14T10:59:09Z)
• Vintix: Action Model via In-Context Reinforcement Learning [72.65703565352769]
  We present the first steps toward scaling ICRL by introducing a fixed, cross-domain model capable of learning behaviors through in-context reinforcement learning. Our results demonstrate that Algorithm Distillation, a framework designed to facilitate ICRL, offers a compelling and competitive alternative to expert distillation to construct versatile action models.
  arXiv  Detail & Related papers  (2025-01-31T18:57:08Z)
• Cognitive LLMs: Towards Integrating Cognitive Architectures and Large Language Models for Manufacturing Decision-making [51.737762570776006]
  LLM-ACTR is a novel neuro-symbolic architecture that provides human-aligned and versatile decision-making. Our framework extracts and embeds knowledge of ACT-R's internal decision-making process as latent neural representations. Our experiments on novel Design for Manufacturing tasks show both improved task performance as well as improved grounded decision-making capability.
  arXiv  Detail & Related papers  (2024-08-17T11:49:53Z)
• Variable-Agnostic Causal Exploration for Reinforcement Learning [56.52768265734155]
  We introduce a novel framework, Variable-Agnostic Causal Exploration for Reinforcement Learning (VACERL) Our approach automatically identifies crucial observation-action steps associated with key variables using attention mechanisms. It constructs the causal graph connecting these steps, which guides the agent towards observation-action pairs with greater causal influence on task completion.
  arXiv  Detail & Related papers  (2024-07-17T09:45:27Z)
• Fine-Grained Causal Dynamics Learning with Quantization for Improving Robustness in Reinforcement Learning [26.34622544479565]
  Causal dynamics learning is a promising approach to enhancing robustness in reinforcement learning. We propose a novel model that infers fine-grained causal structures and employs them for prediction.
  arXiv  Detail & Related papers  (2024-06-05T13:13:58Z)
• Multi-modal Causal Structure Learning and Root Cause Analysis [67.67578590390907]
  We propose Mulan, a unified multi-modal causal structure learning method for root cause localization. We leverage a log-tailored language model to facilitate log representation learning, converting log sequences into time-series data. We also introduce a novel key performance indicator-aware attention mechanism for assessing modality reliability and co-learning a final causal graph.
  arXiv  Detail & Related papers  (2024-02-04T05:50:38Z)
• Causal Structure Learning with Recommendation System [46.90516308311924]
  We first formulate the underlying causal mechanism as a causal structural model and describe a general causal structure learning framework grounded in the real-world working mechanism of recommendation systems. We then derive the learning objective from our framework and propose an augmented Lagrangian solver for efficient optimization.
  arXiv  Detail & Related papers  (2022-10-19T02:31:47Z)
• Generalizing Goal-Conditioned Reinforcement Learning with Variational Causal Reasoning [24.09547181095033]
  Causal Graph is a structure built upon the relation between objects and events. We propose a framework with theoretical performance guarantees that alternates between two steps. Our performance improvement is attributed to the virtuous cycle of causal discovery, transition modeling, and policy training.
  arXiv  Detail & Related papers  (2022-07-19T05:31:16Z)
• A Meta-Reinforcement Learning Algorithm for Causal Discovery [3.4806267677524896]
  Causal structures can enable models to go beyond pure correlation-based inference. Finding causal structures from data poses a significant challenge both in computational effort and accuracy. We develop a meta-reinforcement learning algorithm that performs causal discovery by learning to perform interventions.
  arXiv  Detail & Related papers  (2022-07-18T09:26:07Z)
• INFOrmation Prioritization through EmPOWERment in Visual Model-Based RL [90.06845886194235]
  We propose a modified objective for model-based reinforcement learning (RL) We integrate a term inspired by variational empowerment into a state-space model based on mutual information. We evaluate the approach on a suite of vision-based robot control tasks with natural video backgrounds.
  arXiv  Detail & Related papers  (2022-04-18T23:09:23Z)
• Confounder Identification-free Causal Visual Feature Learning [84.28462256571822]
  We propose a novel Confounder Identification-free Causal Visual Feature Learning (CICF) method, which obviates the need for identifying confounders. CICF models the interventions among different samples based on front-door criterion, and then approximates the global-scope intervening effect upon the instance-level interventions. We uncover the relation between CICF and the popular meta-learning strategy MAML, and provide an interpretation of why MAML works from the theoretical perspective.
  arXiv  Detail & Related papers  (2021-11-26T10:57:47Z)
• Causal Reinforcement Learning using Observational and Interventional Data [14.856472820492364]
  Learning efficiently a causal model of the environment is a key challenge of model RL agents operating in POMDPs. We consider a scenario where the learning agent has the ability to collect online experiences through direct interactions with the environment. We then ask the following questions: can the online and offline experiences be safely combined for learning a causal model.
  arXiv  Detail & Related papers  (2021-06-28T06:58:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.