Local and Global Explanations of Agent Behavior: Integrating Strategy
Summaries with Saliency Maps
- URL: http://arxiv.org/abs/2005.08874v3
- Date: Fri, 29 May 2020 17:54:59 GMT
- Title: Local and Global Explanations of Agent Behavior: Integrating Strategy
Summaries with Saliency Maps
- Authors: Tobias Huber, Katharina Weitz, Elisabeth Andr\'e, Ofra Amir
- Abstract summary: We combine global and local explanations for reinforcement learning agents.
We augment strategy summaries that extract important trajectories of states from simulations with saliency maps.
We find mixed results with respect to augmenting demonstrations with saliency maps.
- Score: 4.568911586155097
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: With advances in reinforcement learning (RL), agents are now being developed
in high-stakes application domains such as healthcare and transportation.
Explaining the behavior of these agents is challenging, as the environments in
which they act have large state spaces, and their decision-making can be
affected by delayed rewards, making it difficult to analyze their behavior. To
address this problem, several approaches have been developed. Some approaches
attempt to convey the $\textit{global}$ behavior of the agent, describing the
actions it takes in different states. Other approaches devised $\textit{local}$
explanations which provide information regarding the agent's decision-making in
a particular state. In this paper, we combine global and local explanation
methods, and evaluate their joint and separate contributions, providing (to the
best of our knowledge) the first user study of combined local and global
explanations for RL agents. Specifically, we augment strategy summaries that
extract important trajectories of states from simulations of the agent with
saliency maps which show what information the agent attends to. Our results
show that the choice of what states to include in the summary (global
information) strongly affects people's understanding of agents: participants
shown summaries that included important states significantly outperformed
participants who were presented with agent behavior in a randomly set of chosen
world-states. We find mixed results with respect to augmenting demonstrations
with saliency maps (local information), as the addition of saliency maps did
not significantly improve performance in most cases. However, we do find some
evidence that saliency maps can help users better understand what information
the agent relies on in its decision making, suggesting avenues for future work
that can further improve explanations of RL agents.
Related papers
- BET: Explaining Deep Reinforcement Learning through The Error-Prone
Decisions [7.139669387895207]
We propose a novel self-interpretable structure, named Backbone Extract Tree (BET), to better explain the agent's behavior.
At a high level, BET hypothesizes that states in which the agent consistently executes uniform decisions exhibit a reduced propensity for errors.
We show BET's superiority over existing self-interpretable models in terms of explanation fidelity.
arXiv Detail & Related papers (2024-01-14T11:45:05Z) - Explaining Reinforcement Learning Agents Through Counterfactual Action
Outcomes [9.108253909440489]
We propose COViz'', a new local explanation method that visually compares the outcome of an agent's chosen action to a counterfactual one.
In contrast to most local explanations that provide state-limited observations of the agent's motivation, our method depicts alternative trajectories the agent could have taken from the given state and their outcomes.
arXiv Detail & Related papers (2023-12-18T11:34:58Z) - Sim-to-Real Causal Transfer: A Metric Learning Approach to
Causally-Aware Interaction Representations [62.48505112245388]
We take an in-depth look at the causal awareness of modern representations of agent interactions.
We show that recent representations are already partially resilient to perturbations of non-causal agents.
We propose a metric learning approach that regularizes latent representations with causal annotations.
arXiv Detail & Related papers (2023-12-07T18:57:03Z) - Information Design in Multi-Agent Reinforcement Learning [61.140924904755266]
Reinforcement learning (RL) is inspired by the way human infants and animals learn from the environment.
Research in computational economics distills two ways to influence others directly: by providing tangible goods (mechanism design) and by providing information (information design)
arXiv Detail & Related papers (2023-05-08T07:52:15Z) - GANterfactual-RL: Understanding Reinforcement Learning Agents'
Strategies through Visual Counterfactual Explanations [0.7874708385247353]
We propose a novel but simple method to generate counterfactual explanations for RL agents.
Our method is fully model-agnostic and we demonstrate that it outperforms the only previous method in several computational metrics.
arXiv Detail & Related papers (2023-02-24T15:29:43Z) - Integrating Policy Summaries with Reward Decomposition for Explaining
Reinforcement Learning Agents [3.8520321531809705]
Methods that help users understand the behavior of such agents can roughly be divided into local explanations and global explanations.
We study a novel combination of local and global explanations for reinforcement learning agents.
arXiv Detail & Related papers (2022-10-21T08:57:46Z) - Experiential Explanations for Reinforcement Learning [15.80179578318569]
Reinforcement Learning systems can be complex and non-interpretable.
We propose a technique, Experiential Explanations, to generate counterfactual explanations.
arXiv Detail & Related papers (2022-10-10T14:27:53Z) - Explaining Reinforcement Learning Policies through Counterfactual
Trajectories [147.7246109100945]
A human developer must validate that an RL agent will perform well at test-time.
Our method conveys how the agent performs under distribution shifts by showing the agent's behavior across a wider trajectory distribution.
In a user study, we demonstrate that our method enables users to score better than baseline methods on one of two agent validation tasks.
arXiv Detail & Related papers (2022-01-29T00:52:37Z) - Modeling Bounded Rationality in Multi-Agent Simulations Using Rationally
Inattentive Reinforcement Learning [85.86440477005523]
We study more human-like RL agents which incorporate an established model of human-irrationality, the Rational Inattention (RI) model.
RIRL models the cost of cognitive information processing using mutual information.
We show that using RIRL yields a rich spectrum of new equilibrium behaviors that differ from those found under rational assumptions.
arXiv Detail & Related papers (2022-01-18T20:54:00Z) - Collective eXplainable AI: Explaining Cooperative Strategies and Agent
Contribution in Multiagent Reinforcement Learning with Shapley Values [68.8204255655161]
This study proposes a novel approach to explain cooperative strategies in multiagent RL using Shapley values.
Results could have implications for non-discriminatory decision making, ethical and responsible AI-derived decisions or policy making under fairness constraints.
arXiv Detail & Related papers (2021-10-04T10:28:57Z) - InfoBot: Transfer and Exploration via the Information Bottleneck [105.28380750802019]
A central challenge in reinforcement learning is discovering effective policies for tasks where rewards are sparsely distributed.
We propose to learn about decision states from prior experience.
We find that this simple mechanism effectively identifies decision states, even in partially observed settings.
arXiv Detail & Related papers (2019-01-30T15:33:58Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.