Towards Causal Credit Assignment

• URL: http://arxiv.org/abs/2212.11636v2
• Date: Wed, 17 May 2023 12:27:43 GMT
• Title: Towards Causal Credit Assignment
• Authors: M\'aty\'as Schubert
• Abstract summary: Hindsight Credit Assignment is a promising, but still unexplored candidate, which aims to solve the problems of both long-term and counterfactual credit assignment. In this thesis, we empirically investigate Hindsight Credit Assignment to identify its main benefits, and key points to improve. We show that our modification greatly decreases the workload of Hindsight Credit Assignment, making it more efficient and enabling it to outperform the baseline credit assignment method on various tasks.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Adequately assigning credit to actions for future outcomes based on their contributions is a long-standing open challenge in Reinforcement Learning. The assumptions of the most commonly used credit assignment method are disadvantageous in tasks where the effects of decisions are not immediately evident. Furthermore, this method can only evaluate actions that have been selected by the agent, making it highly inefficient. Still, no alternative methods have been widely adopted in the field. Hindsight Credit Assignment is a promising, but still unexplored candidate, which aims to solve the problems of both long-term and counterfactual credit assignment. In this thesis, we empirically investigate Hindsight Credit Assignment to identify its main benefits, and key points to improve. Then, we apply it to factored state representations, and in particular to state representations based on the causal structure of the environment. In this setting, we propose a variant of Hindsight Credit Assignment that effectively exploits a given causal structure. We show that our modification greatly decreases the workload of Hindsight Credit Assignment, making it more efficient and enabling it to outperform the baseline credit assignment method on various tasks. This opens the way to other methods based on given or learned causal structures.

Related papers

• A Survey of Temporal Credit Assignment in Deep Reinforcement Learning [47.17998784925718]
  The Credit Assignment Problem (CAP) refers to the longstanding challenge of Reinforcement Learning (RL) agents to associate actions with their long-term consequences. We propose a unifying formalism for credit that enables equitable comparisons of state-of-the-art algorithms. We discuss the challenges posed by delayed effects, transpositions, and a lack of action influence, and analyse how existing methods aim to address them.
  arXiv  Detail & Related papers  (2023-12-02T08:49:51Z)
• Would I have gotten that reward? Long-term credit assignment by counterfactual contribution analysis [50.926791529605396]
  We introduce Counterfactual Contribution Analysis (COCOA), a new family of model-based credit assignment algorithms. Our algorithms achieve precise credit assignment by measuring the contribution of actions upon obtaining subsequent rewards.
  arXiv  Detail & Related papers  (2023-06-29T09:27:27Z)
• Selective Credit Assignment [57.41789233550586]
  We describe a unified view on temporal-difference algorithms for selective credit assignment. We present insights into applying weightings to value-based learning and planning algorithms.
  arXiv  Detail & Related papers  (2022-02-20T00:07:57Z)
• Direct Advantage Estimation [63.52264764099532]
  We show that the expected return may depend on the policy in an undesirable way which could slow down learning. We propose the Direct Advantage Estimation (DAE), a novel method that can model the advantage function and estimate it directly from data. If desired, value functions can also be seamlessly integrated into DAE and be updated in a similar way to Temporal Difference Learning.
  arXiv  Detail & Related papers  (2021-09-13T16:09:31Z)
• An Information-Theoretic Perspective on Credit Assignment in Reinforcement Learning [14.367867691822026]
  We argue that it is not the sparsity of the reward itself that causes difficulty in credit assignment, but rather the emph information sparsity We outline several information-theoretic mechanisms for measuring credit under a fixed behavior policy, highlighting the potential of information theory as a key tool towards provably-efficient credit assignment.
  arXiv  Detail & Related papers  (2021-03-10T17:50:15Z)
• Explanations of Machine Learning predictions: a mandatory step for its application to Operational Processes [61.20223338508952]
  Credit Risk Modelling plays a paramount role. Recent machine and deep learning techniques have been applied to the task. We suggest to use LIME technique to tackle the explainability problem in this field.
  arXiv  Detail & Related papers  (2020-12-30T10:27:59Z)
• Explainable AI for Interpretable Credit Scoring [0.8379286663107844]
  Credit scoring helps financial experts make better decisions regarding whether or not to accept a loan application. Regulations have added the need for model interpretability to ensure that algorithmic decisions are understandable coherent. We present a credit scoring model that is both accurate and interpretable.
  arXiv  Detail & Related papers  (2020-12-03T18:44:03Z)
• Counterfactual Credit Assignment in Model-Free Reinforcement Learning [47.79277857377155]
  Credit assignment in reinforcement learning is the problem of measuring an action's influence on future rewards. We adapt the notion of counterfactuals from causality theory to a model-free RL setup. We formulate a family of policy algorithms that use future-conditional value functions as baselines or critics, and show that they are provably low variance.
  arXiv  Detail & Related papers  (2020-11-18T18:41:44Z)
• Intelligent Credit Limit Management in Consumer Loans Based on Causal Inference [5.292270534252169]
  Credit limits are adjusted based on limited strategies, which are developed by experienced professionals. In this paper, we present a data-driven approach to manage the credit limit intelligently.
  arXiv  Detail & Related papers  (2020-07-10T06:22:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.