State2Explanation: Concept-Based Explanations to Benefit Agent Learning and User Understanding

• URL: http://arxiv.org/abs/2309.12482v2
• Date: Fri, 10 Nov 2023 16:51:11 GMT
• Title: State2Explanation: Concept-Based Explanations to Benefit Agent Learning and User Understanding
• Authors: Devleena Das, Sonia Chernova, Been Kim
• Abstract summary: We contribute a desiderata for defining concepts in sequential decision making settings. We explore how concept-based explanations of an RL agent's decision making can improve the agent's learning rate. We contribute a unified framework, State2Explanation (S2E), that involves learning a joint embedding model between state-action pairs.
• Score: 15.503872709445249
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As more non-AI experts use complex AI systems for daily tasks, there has been an increasing effort to develop methods that produce explanations of AI decision making that are understandable by non-AI experts. Towards this effort, leveraging higher-level concepts and producing concept-based explanations have become a popular method. Most concept-based explanations have been developed for classification techniques, and we posit that the few existing methods for sequential decision making are limited in scope. In this work, we first contribute a desiderata for defining concepts in sequential decision making settings. Additionally, inspired by the Protege Effect which states explaining knowledge often reinforces one's self-learning, we explore how concept-based explanations of an RL agent's decision making can in turn improve the agent's learning rate, as well as improve end-user understanding of the agent's decision making. To this end, we contribute a unified framework, State2Explanation (S2E), that involves learning a joint embedding model between state-action pairs and concept-based explanations, and leveraging such learned model to both (1) inform reward shaping during an agent's training, and (2) provide explanations to end-users at deployment for improved task performance. Our experimental validations, in Connect 4 and Lunar Lander, demonstrate the success of S2E in providing a dual-benefit, successfully informing reward shaping and improving agent learning rate, as well as significantly improving end user task performance at deployment time.

Related papers

• Semifactual Explanations for Reinforcement Learning [1.5320737596132754]
  Reinforcement Learning (RL) is a learning paradigm in which the agent learns from its environment through trial and error. Deep reinforcement learning (DRL) algorithms represent the agent's policies using neural networks, making their decisions difficult to interpret. Explaining the behaviour of DRL agents is necessary to advance user trust, increase engagement, and facilitate integration with real-life tasks.
  arXiv  Detail & Related papers  (2024-09-09T08:37:47Z)
• Explaining Explainability: Towards Deeper Actionable Insights into Deep Learning through Second-order Explainability [70.60433013657693]
  Second-order explainable AI (SOXAI) was recently proposed to extend explainable AI (XAI) from the instance level to the dataset level. We demonstrate for the first time, via example classification and segmentation cases, that eliminating irrelevant concepts from the training set based on actionable insights from SOXAI can enhance a model's performance.
  arXiv  Detail & Related papers  (2023-06-14T23:24:01Z)
• Distinguish Before Answer: Generating Contrastive Explanation as Knowledge for Commonsense Question Answering [61.53454387743701]
  We propose CPACE, a concept-centric Prompt-bAsed Contrastive Explanation Generation model. CPACE converts obtained symbolic knowledge into a contrastive explanation for better distinguishing the differences among given candidates. We conduct a series of experiments on three widely-used question-answering datasets: CSQA, QASC, and OBQA.
  arXiv  Detail & Related papers  (2023-05-14T12:12:24Z)
• Towards Human Cognition Level-based Experiment Design for Counterfactual Explanations (XAI) [68.8204255655161]
  The emphasis of XAI research appears to have turned to a more pragmatic explanation approach for better understanding. An extensive area where cognitive science research may substantially influence XAI advancements is evaluating user knowledge and feedback. We propose a framework to experiment with generating and evaluating the explanations on the grounds of different cognitive levels of understanding.
  arXiv  Detail & Related papers  (2022-10-31T19:20:22Z)
• Redefining Counterfactual Explanations for Reinforcement Learning: Overview, Challenges and Opportunities [2.0341936392563063]
  Most explanation methods for AI are focused on developers and expert users. Counterfactual explanations offer users advice on what can be changed in the input for the output of the black-box model to change. Counterfactuals are user-friendly and provide actionable advice for achieving the desired output from the AI system.
  arXiv  Detail & Related papers  (2022-10-21T09:50:53Z)
• Inverse Online Learning: Understanding Non-Stationary and Reactionary Policies [79.60322329952453]
  We show how to develop interpretable representations of how agents make decisions. By understanding the decision-making processes underlying a set of observed trajectories, we cast the policy inference problem as the inverse to this online learning problem. We introduce a practical algorithm for retrospectively estimating such perceived effects, alongside the process through which agents update them. Through application to the analysis of UNOS organ donation acceptance decisions, we demonstrate that our approach can bring valuable insights into the factors that govern decision processes and how they change over time.
  arXiv  Detail & Related papers  (2022-03-14T17:40:42Z)
• Abstraction, Validation, and Generalization for Explainable Artificial Intelligence [5.142415132534397]
  Methods to explain AI have been proposed to answer this challenge, but a lack of theory impedes the development of systematic abstractions. We propose Bayesian Teaching as a framework for unifying explainable AI (XAI) by integrating machine learning and human learning.
  arXiv  Detail & Related papers  (2021-05-16T20:40:23Z)
• Weakly Supervised Multi-task Learning for Concept-based Explainability [3.441021278275805]
  We leverage multi-task learning to train a neural network that jointly learns to predict a decision task. There are two main challenges to overcome: concept label scarcity and the joint learning. We show it is possible to improve performance at both tasks by combining labels of heterogeneous quality.
  arXiv  Detail & Related papers  (2021-04-26T10:42:19Z)
• What Did You Think Would Happen? Explaining Agent Behaviour Through Intended Outcomes [30.056732656973637]
  We present a novel form of explanation for Reinforcement Learning, based around the notion of intended outcome. These explanations describe the outcome an agent is trying to achieve by its actions. We provide a simple proof that general methods for post-hoc explanations of this nature are impossible in traditional reinforcement learning.
  arXiv  Detail & Related papers  (2020-11-10T12:05:08Z)
• A Competence-aware Curriculum for Visual Concepts Learning via Question Answering [95.35905804211698]
  We propose a competence-aware curriculum for visual concept learning in a question-answering manner. We design a neural-symbolic concept learner for learning the visual concepts and a multi-dimensional Item Response Theory (mIRT) model for guiding the learning process. Experimental results on CLEVR show that with a competence-aware curriculum, the proposed method achieves state-of-the-art performances.
  arXiv  Detail & Related papers  (2020-07-03T05:08:09Z)
• A general framework for scientifically inspired explanations in AI [76.48625630211943]
  We instantiate the concept of structure of scientific explanation as the theoretical underpinning for a general framework in which explanations for AI systems can be implemented. This framework aims to provide the tools to build a "mental-model" of any AI system so that the interaction with the user can provide information on demand and be closer to the nature of human-made explanations.
  arXiv  Detail & Related papers  (2020-03-02T10:32:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.