Causal Influence Detection for Improving Efficiency in Reinforcement Learning

• URL: http://arxiv.org/abs/2106.03443v1
• Date: Mon, 7 Jun 2021 09:21:56 GMT
• Title: Causal Influence Detection for Improving Efficiency in Reinforcement Learning
• Authors: Maximilian Seitzer and Bernhard Sch\"olkopf and Georg Martius
• Abstract summary: We introduce a measure of situation-dependent causal influence based on conditional mutual information. We show that it can reliably detect states of influence. All modified algorithms show strong increases in data efficiency on robotic manipulation tasks.
• Score: 11.371889042789219
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Many reinforcement learning (RL) environments consist of independent entities that interact sparsely. In such environments, RL agents have only limited influence over other entities in any particular situation. Our idea in this work is that learning can be efficiently guided by knowing when and what the agent can influence with its actions. To achieve this, we introduce a measure of situation-dependent causal influence based on conditional mutual information and show that it can reliably detect states of influence. We then propose several ways to integrate this measure into RL algorithms to improve exploration and off-policy learning. All modified algorithms show strong increases in data efficiency on robotic manipulation tasks.

Related papers

• Variable-Agnostic Causal Exploration for Reinforcement Learning [56.52768265734155]
  We introduce a novel framework, Variable-Agnostic Causal Exploration for Reinforcement Learning (VACERL) Our approach automatically identifies crucial observation-action steps associated with key variables using attention mechanisms. It constructs the causal graph connecting these steps, which guides the agent towards observation-action pairs with greater causal influence on task completion.
  arXiv  Detail & Related papers  (2024-07-17T09:45:27Z)
• External Model Motivated Agents: Reinforcement Learning for Enhanced Environment Sampling [3.536024441537599]
  Unlike reinforcement learning (RL) agents, humans remain capable multitaskers in changing environments. We propose an agent influence framework for RL agents to improve the adaptation efficiency of external models in changing environments. Our results show that our method outperforms the baselines in terms of external model adaptation on metrics that measure both efficiency and performance.
  arXiv  Detail & Related papers  (2024-06-28T23:31:22Z)
• Causal Coordinated Concurrent Reinforcement Learning [8.654978787096807]
  We propose a novel algorithmic framework for data sharing and coordinated exploration for the purpose of learning more data-efficient and better performing policies under a concurrent reinforcement learning setting. Our algorithm leverages a causal inference algorithm in the form of Additive Noise Model - Mixture Model (ANM-MM) in extracting model parameters governing individual differentials via independence enforcement. We propose a new data sharing scheme based on a similarity measure of the extracted model parameters and demonstrate superior learning speeds on a set of autoregressive, pendulum and cart-pole swing-up tasks.
  arXiv  Detail & Related papers  (2024-01-31T17:20:28Z)
• Inapplicable Actions Learning for Knowledge Transfer in Reinforcement Learning [3.194414753332705]
  We show that learning inapplicable actions greatly improves the sample efficiency of RL algorithms. Thanks to the transferability of the knowledge acquired, it can be reused in other tasks and domains to make the learning process more efficient.
  arXiv  Detail & Related papers  (2022-11-28T17:45:39Z)
• Flexible Attention-Based Multi-Policy Fusion for Efficient Deep Reinforcement Learning [78.31888150539258]
  Reinforcement learning (RL) agents have long sought to approach the efficiency of human learning. Prior studies in RL have incorporated external knowledge policies to help agents improve sample efficiency. We present Knowledge-Grounded RL (KGRL), an RL paradigm fusing multiple knowledge policies and aiming for human-like efficiency and flexibility.
  arXiv  Detail & Related papers  (2022-10-07T17:56:57Z)
• Denoised MDPs: Learning World Models Better Than the World Itself [94.74665254213588]
  This work categorizes information out in the wild into four types based on controllability and relation with reward, and formulates useful information as that which is both controllable and reward-relevant. Experiments on variants of DeepMind Control Suite and RoboDesk demonstrate superior performance of our denoised world model over using raw observations alone.
  arXiv  Detail & Related papers  (2022-06-30T17:59:49Z)
• Improving Robustness of Learning-based Autonomous Steering Using Adversarial Images [58.287120077778205]
  We introduce a framework for analyzing robustness of the learning algorithm w.r.t varying quality in the image input for autonomous driving. Using the results of sensitivity analysis, we propose an algorithm to improve the overall performance of the task of "learning to steer"
  arXiv  Detail & Related papers  (2021-02-26T02:08:07Z)
• Disturbing Reinforcement Learning Agents with Corrupted Rewards [62.997667081978825]
  We analyze the effects of different attack strategies based on reward perturbations on reinforcement learning algorithms. We show that smoothly crafting adversarial rewards are able to mislead the learner, and that using low exploration probability values, the policy learned is more robust to corrupt rewards.
  arXiv  Detail & Related papers  (2021-02-12T15:53:48Z)
• Discrete Action On-Policy Learning with Action-Value Critic [72.20609919995086]
  Reinforcement learning (RL) in discrete action space is ubiquitous in real-world applications, but its complexity grows exponentially with the action-space dimension. We construct a critic to estimate action-value functions, apply it on correlated actions, and combine these critic estimated action values to control the variance of gradient estimation. These efforts result in a new discrete action on-policy RL algorithm that empirically outperforms related on-policy algorithms relying on variance control techniques.
  arXiv  Detail & Related papers  (2020-02-10T04:23:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.