Identifying Reasoning Flaws in Planning-Based RL Using Tree Explanations

• URL: http://arxiv.org/abs/2109.13978v1
• Date: Tue, 28 Sep 2021 18:39:03 GMT
• Title: Identifying Reasoning Flaws in Planning-Based RL Using Tree Explanations
• Authors: Kin-Ho Lam, Zhengxian Lin, Jed Irvine, Jonathan Dodge, Zeyad T Shureih, Roli Khanna, Minsuk Kahng, Alan Fern
• Abstract summary: We consider identifying flaws in a planning-based deep reinforcement learning agent for a real-time strategy game. This gives the potential for humans to identify flaws at the level of reasoning steps in the tree. It is unclear whether humans will be able to identify such flaws due to the size and complexity of trees.
• Score: 16.610062357578283
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Enabling humans to identify potential flaws in an agent's decision making is an important Explainable AI application. We consider identifying such flaws in a planning-based deep reinforcement learning (RL) agent for a complex real-time strategy game. In particular, the agent makes decisions via tree search using a learned model and evaluation function over interpretable states and actions. This gives the potential for humans to identify flaws at the level of reasoning steps in the tree, even if the entire reasoning process is too complex to understand. However, it is unclear whether humans will be able to identify such flaws due to the size and complexity of trees. We describe a user interface and case study, where a small group of AI experts and developers attempt to identify reasoning flaws due to inaccurate agent learning. Overall, the interface allowed the group to identify a number of significant flaws of varying types, demonstrating the promise of this approach.

Related papers

• GPTree: Towards Explainable Decision-Making via LLM-powered Decision Trees [0.0]
  GPTree is a novel framework combining explainability of decision trees with the advanced reasoning capabilities of LLMs. Our decision tree achieved a 7.8% precision rate for identifying "unicorn" startups at the inception stage of a startup.
  arXiv  Detail & Related papers  (2024-11-13T00:14:09Z)
• Pangu-Agent: A Fine-Tunable Generalist Agent with Structured Reasoning [50.47568731994238]
  Key method for creating Artificial Intelligence (AI) agents is Reinforcement Learning (RL) This paper presents a general framework model for integrating and learning structured reasoning into AI agents' policies.
  arXiv  Detail & Related papers  (2023-12-22T17:57:57Z)
• Evaluating the Utility of Model Explanations for Model Development [54.23538543168767]
  We evaluate whether explanations can improve human decision-making in practical scenarios of machine learning model development. To our surprise, we did not find evidence of significant improvement on tasks when users were provided with any of the saliency maps. These findings suggest caution regarding the usefulness and potential for misunderstanding in saliency-based explanations.
  arXiv  Detail & Related papers  (2023-12-10T23:13:23Z)
• Probabilistic Tree-of-thought Reasoning for Answering Knowledge-intensive Complex Questions [93.40614719648386]
  Large language models (LLMs) are capable of answering knowledge-intensive complex questions with chain-of-thought (CoT) reasoning. Recent works turn to retrieving external knowledge to augment CoT reasoning. We propose a novel approach: Probabilistic Tree-of-thought Reasoning (ProbTree)
  arXiv  Detail & Related papers  (2023-11-23T12:52:37Z)
• A Closer Look at the Self-Verification Abilities of Large Language Models in Logical Reasoning [73.77088902676306]
  We take a closer look at the self-verification abilities of large language models (LLMs) in the context of logical reasoning. Our main findings suggest that existing LLMs could struggle to identify fallacious reasoning steps accurately and may fall short of guaranteeing the validity of self-verification methods.
  arXiv  Detail & Related papers  (2023-11-14T07:13:10Z)
• Explaining Tree Model Decisions in Natural Language for Network Intrusion Detection [18.5400518912912]
  Network intrusion detection (NID) systems which leverage machine learning have been shown to have strong performance in practice when used to detect malicious network traffic. Decision trees in particular offer a strong balance between performance and simplicity, but require users of NID systems to have background knowledge in machine learning to interpret. In this work, we explore the use of large language models (LLMs) to provide explanations and additional background knowledge for decision tree NID systems.
  arXiv  Detail & Related papers  (2023-10-30T15:40:34Z)
• Limits of Actor-Critic Algorithms for Decision Tree Policies Learning in IBMDPs [9.587070290189507]
  Interpretability of AI models allows for user safety checks to build trust in such AIs. Decision Trees (DTs) provide a global look at the learned model and transparently reveal which features of the input are critical for making a decision. Recent Reinforcement Learning framework has been proposed to explore the space of DTs using deep RL.
  arXiv  Detail & Related papers  (2023-09-23T13:06:20Z)
• Interpretability at Scale: Identifying Causal Mechanisms in Alpaca [62.65877150123775]
  We use Boundless DAS to efficiently search for interpretable causal structure in large language models while they follow instructions. Our findings mark a first step toward faithfully understanding the inner-workings of our ever-growing and most widely deployed language models.
  arXiv  Detail & Related papers  (2023-05-15T17:15:40Z)
• On Explainability in AI-Solutions: A Cross-Domain Survey [4.394025678691688]
  In automatically deriving a system model, AI algorithms learn relations in data that are not detectable for humans. The more complex a model, the more difficult it is for a human to understand the reasoning for the decisions. This work provides an extensive survey of literature on this topic, which, to a large part, consists of other surveys.
  arXiv  Detail & Related papers  (2022-10-11T06:21:47Z)
• Neural Causal Models for Counterfactual Identification and Estimation [62.30444687707919]
  We study the evaluation of counterfactual statements through neural models. First, we show that neural causal models (NCMs) are expressive enough. Second, we develop an algorithm for simultaneously identifying and estimating counterfactual distributions.
  arXiv  Detail & Related papers  (2022-09-30T18:29:09Z)
• An Objective Metric for Explainable AI: How and Why to Estimate the Degree of Explainability [3.04585143845864]
  We present a new model-agnostic metric to measure the Degree of eXplainability of correct information in an objective way. We designed a few experiments and a user-study on two realistic AI-based systems for healthcare and finance.
  arXiv  Detail & Related papers  (2021-09-11T17:44:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.