Local Explanations for Reinforcement Learning
- URL: http://arxiv.org/abs/2202.03597v1
- Date: Tue, 8 Feb 2022 02:02:09 GMT
- Title: Local Explanations for Reinforcement Learning
- Authors: Ronny Luss, Amit Dhurandhar, Miao Liu
- Abstract summary: We propose a novel perspective to understanding RL policies based on identifying important states from automatically learned meta-states.
We show that our algorithm to find meta-states converges and the objective that selects important states from each meta-state is submodular leading to efficient high quality greedy selection.
- Score: 14.87922813917482
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Many works in explainable AI have focused on explaining black-box
classification models. Explaining deep reinforcement learning (RL) policies in
a manner that could be understood by domain users has received much less
attention. In this paper, we propose a novel perspective to understanding RL
policies based on identifying important states from automatically learned
meta-states. The key conceptual difference between our approach and many
previous ones is that we form meta-states based on locality governed by the
expert policy dynamics rather than based on similarity of actions, and that we
do not assume any particular knowledge of the underlying topology of the state
space. Theoretically, we show that our algorithm to find meta-states converges
and the objective that selects important states from each meta-state is
submodular leading to efficient high quality greedy selection. Experiments on
four domains (four rooms, door-key, minipacman, and pong) and a carefully
conducted user study illustrate that our perspective leads to better
understanding of the policy. We conjecture that this is a result of our
meta-states being more intuitive in that the corresponding important states are
strong indicators of tractable intermediate goals that are easier for humans to
interpret and follow.
Related papers
- MaxMI: A Maximal Mutual Information Criterion for Manipulation Concept Discovery [8.98306885751389]
We introduce an information-theoretic criterion to characterize the regularities that signify a set of physical states.
We develop a framework that trains a concept discovery network using this criterion.
arXiv Detail & Related papers (2024-07-21T07:56:48Z) - Learning Interpretable Policies in Hindsight-Observable POMDPs through
Partially Supervised Reinforcement Learning [57.67629402360924]
We introduce the Partially Supervised Reinforcement Learning (PSRL) framework.
At the heart of PSRL is the fusion of both supervised and unsupervised learning.
We show that PSRL offers a potent balance, enhancing model interpretability while preserving, and often significantly outperforming, the performance benchmarks set by traditional methods.
arXiv Detail & Related papers (2024-02-14T16:23:23Z) - Towards an Information Theoretic Framework of Context-Based Offline
Meta-Reinforcement Learning [50.976910714839065]
Context-based OMRL (COMRL) as a popular paradigm, aims to learn a universal policy conditioned on effective task representations.
We show that COMRL algorithms are essentially optimizing the same mutual information objective between the task variable $boldsymbolM$ and its latent representation $boldsymbolZ$ by implementing various approximate bounds.
Based on the theoretical insight and the information bottleneck principle, we arrive at a novel algorithm dubbed UNICORN, which exhibits remarkable generalization across a broad spectrum of RL benchmarks.
arXiv Detail & Related papers (2024-02-04T09:58:42Z) - State Representation Learning for Goal-Conditioned Reinforcement
Learning [9.162936410696407]
This paper presents a novel state representation for reward-free Markov decision processes.
The idea is to learn, in a self-supervised manner, an embedding space where between pairs of embedded states correspond to the minimum number of actions needed to transition between them.
We show how this representation can be leveraged to learn goal-conditioned policies.
arXiv Detail & Related papers (2022-05-04T09:20:09Z) - Interpretable Reinforcement Learning with Multilevel Subgoal Discovery [77.34726150561087]
We propose a novel Reinforcement Learning model for discrete environments.
In the model, an agent learns information about environment in the form of probabilistic rules.
No reward function is required for learning; an agent only needs to be given a primary goal to achieve.
arXiv Detail & Related papers (2022-02-15T14:04:44Z) - Wasserstein Distance Maximizing Intrinsic Control [14.963071654271756]
This paper deals with the problem of learning a skill-conditioned policy that acts meaningfully in the absence of a reward signal.
It shows that such an objective leads to a policy that covers more distance in the MDP than diversity based objectives.
arXiv Detail & Related papers (2021-10-28T17:46:07Z) - DisTop: Discovering a Topological representation to learn diverse and
rewarding skills [0.0]
DisTop is a new model that simultaneously learns diverse skills and focuses on improving rewarding skills.
DisTop builds a discrete topology of the environment using an unsupervised contrastive loss, a growing network and a goal-conditioned policy.
We show that DisTop achieves state-of-the-art performance in comparison with hierarchical reinforcement learning (HRL) when rewards are sparse.
arXiv Detail & Related papers (2021-06-06T10:09:05Z) - Learning for Visual Navigation by Imagining the Success [66.99810227193196]
We propose to learn to imagine a latent representation of the successful (sub-)goal state.
ForeSIT is trained to imagine the recurrent latent representation of a future state that leads to success.
We develop an efficient learning algorithm to train ForeSIT in an on-policy manner and integrate it into our RL objective.
arXiv Detail & Related papers (2021-02-28T10:25:46Z) - Explainable Deep Classification Models for Domain Generalization [94.43131722655617]
Explanations are defined as regions of visual evidence upon which a deep classification network makes a decision.
Our training strategy enforces a periodic saliency-based feedback to encourage the model to focus on the image regions that directly correspond to the ground-truth object.
arXiv Detail & Related papers (2020-03-13T22:22:15Z) - InfoBot: Transfer and Exploration via the Information Bottleneck [105.28380750802019]
A central challenge in reinforcement learning is discovering effective policies for tasks where rewards are sparsely distributed.
We propose to learn about decision states from prior experience.
We find that this simple mechanism effectively identifies decision states, even in partially observed settings.
arXiv Detail & Related papers (2019-01-30T15:33:58Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.