Machine versus Human Attention in Deep Reinforcement Learning Tasks
- URL: http://arxiv.org/abs/2010.15942v3
- Date: Tue, 2 Nov 2021 23:01:39 GMT
- Title: Machine versus Human Attention in Deep Reinforcement Learning Tasks
- Authors: Sihang Guo, Ruohan Zhang, Bo Liu, Yifeng Zhu, Mary Hayhoe, Dana
Ballard, Peter Stone
- Abstract summary: We shed light on the inner workings of such trained models by analyzing the pixels that they attend to during task execution.
We compare the saliency maps of RL agents against visual attention models of human experts when learning to play Atari games.
- Score: 38.80270891345248
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Deep reinforcement learning (RL) algorithms are powerful tools for solving
visuomotor decision tasks. However, the trained models are often difficult to
interpret, because they are represented as end-to-end deep neural networks. In
this paper, we shed light on the inner workings of such trained models by
analyzing the pixels that they attend to during task execution, and comparing
them with the pixels attended to by humans executing the same tasks. To this
end, we investigate the following two questions that, to the best of our
knowledge, have not been previously studied. 1) How similar are the visual
representations learned by RL agents and humans when performing the same task?
and, 2) How do similarities and differences in these learned representations
explain RL agents' performance on these tasks? Specifically, we compare the
saliency maps of RL agents against visual attention models of human experts
when learning to play Atari games. Further, we analyze how hyperparameters of
the deep RL algorithm affect the learned representations and saliency maps of
the trained agents. The insights provided have the potential to inform novel
algorithms for closing the performance gap between human experts and RL agents.
Related papers
- ViSaRL: Visual Reinforcement Learning Guided by Human Saliency [6.969098096933547]
We introduce Visual Saliency-Guided Reinforcement Learning (ViSaRL)
Using ViSaRL to learn visual representations significantly improves the success rate, sample efficiency, and generalization of an RL agent.
We show that visual representations learned using ViSaRL are robust to various sources of visual perturbations including perceptual noise and scene variations.
arXiv Detail & Related papers (2024-03-16T14:52:26Z) - Diagnosing and exploiting the computational demands of videos games for
deep reinforcement learning [13.98405611352641]
We introduce the Learning Challenge Diagnosticator (LCD), a tool that measures the perceptual and reinforcement learning demands of a task.
We use LCD to discover a novel taxonomy of challenges in the Procgen benchmark, and demonstrate that these predictions are both highly reliable and can instruct algorithmic development.
arXiv Detail & Related papers (2023-09-22T21:03:33Z) - Leveraging Reward Consistency for Interpretable Feature Discovery in
Reinforcement Learning [69.19840497497503]
It is argued that the commonly used action matching principle is more like an explanation of deep neural networks (DNNs) than the interpretation of RL agents.
We propose to consider rewards, the essential objective of RL agents, as the essential objective of interpreting RL agents.
We verify and evaluate our method on the Atari 2600 games as well as Duckietown, a challenging self-driving car simulator environment.
arXiv Detail & Related papers (2023-09-04T09:09:54Z) - Int-HRL: Towards Intention-based Hierarchical Reinforcement Learning [23.062590084580542]
Int-HRL: Hierarchical RL with intention-based sub-goals that are inferred from human eye gaze.
Our evaluations show that replacing hand-crafted sub-goals with automatically extracted intentions leads to a HRL agent that is significantly more sample efficient than previous methods.
arXiv Detail & Related papers (2023-06-20T12:12:16Z) - Accelerating exploration and representation learning with offline
pre-training [52.6912479800592]
We show that exploration and representation learning can be improved by separately learning two different models from a single offline dataset.
We show that learning a state representation using noise-contrastive estimation and a model of auxiliary reward can significantly improve the sample efficiency on the challenging NetHack benchmark.
arXiv Detail & Related papers (2023-03-31T18:03:30Z) - Agent-Controller Representations: Principled Offline RL with Rich
Exogenous Information [49.06422815335159]
Learning to control an agent from data collected offline is vital for real-world applications of reinforcement learning (RL)
This paper introduces offline RL benchmarks offering the ability to study this problem.
We find that contemporary representation learning techniques can fail on datasets where the noise is a complex and time dependent process.
arXiv Detail & Related papers (2022-10-31T22:12:48Z) - Contrastive Learning as Goal-Conditioned Reinforcement Learning [147.28638631734486]
In reinforcement learning (RL), it is easier to solve a task if given a good representation.
While deep RL should automatically acquire such good representations, prior work often finds that learning representations in an end-to-end fashion is unstable.
We show (contrastive) representation learning methods can be cast as RL algorithms in their own right.
arXiv Detail & Related papers (2022-06-15T14:34:15Z) - Are Gradient-based Saliency Maps Useful in Deep Reinforcement Learning? [4.254099382808598]
Deep Reinforcement Learning (DRL) connects the classic Reinforcement Learning algorithms with Deep Neural Networks.
A problem in DRL is that CNNs are black-boxes and it is hard to understand the decision-making process of agents.
This work brings some of the best-known visualization methods from the field of image classification to the area of Deep Reinforcement Learning.
arXiv Detail & Related papers (2020-12-02T15:38:36Z) - Explainability in Deep Reinforcement Learning [68.8204255655161]
We review recent works in the direction to attain Explainable Reinforcement Learning (XRL)
In critical situations where it is essential to justify and explain the agent's behaviour, better explainability and interpretability of RL models could help gain scientific insight on the inner workings of what is still considered a black box.
arXiv Detail & Related papers (2020-08-15T10:11:42Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.