Matching Representations of Explainable Artificial Intelligence and Eye
Gaze for Human-Machine Interaction
- URL: http://arxiv.org/abs/2102.00179v1
- Date: Sat, 30 Jan 2021 07:42:56 GMT
- Title: Matching Representations of Explainable Artificial Intelligence and Eye
Gaze for Human-Machine Interaction
- Authors: Tiffany Hwu, Mia Levy, Steven Skorheim, David Huber
- Abstract summary: Rapid non-verbal communication of task-based stimuli is a challenge in human-machine teaming.
In this work, we examine the correlations between visual heatmap explanations of a neural network trained to predict driving behavior and eye gaze heatmaps of human drivers.
- Score: 0.7742297876120561
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Rapid non-verbal communication of task-based stimuli is a challenge in
human-machine teaming, particularly in closed-loop interactions such as
driving. To achieve this, we must understand the representations of information
for both the human and machine, and determine a basis for bridging these
representations. Techniques of explainable artificial intelligence (XAI) such
as layer-wise relevance propagation (LRP) provide visual heatmap explanations
for high-dimensional machine learning techniques such as deep neural networks.
On the side of human cognition, visual attention is driven by the bottom-up and
top-down processing of sensory input related to the current task. Since both
XAI and human cognition should focus on task-related stimuli, there may be
overlaps between their representations of visual attention, potentially
providing a means of nonverbal communication between the human and machine. In
this work, we examine the correlations between LRP heatmap explanations of a
neural network trained to predict driving behavior and eye gaze heatmaps of
human drivers. The analysis is used to determine the feasibility of using such
a technique for enhancing driving performance. We find that LRP heatmaps show
increasing levels of similarity with eye gaze according to the task specificity
of the neural network. We then propose how these findings may assist humans by
visually directing attention towards relevant areas. To our knowledge, our work
provides the first known analysis of LRP and eye gaze for driving tasks.
Related papers
- Unidirectional brain-computer interface: Artificial neural network
encoding natural images to fMRI response in the visual cortex [12.1427193917406]
We propose an artificial neural network dubbed VISION to mimic the human brain and show how it can foster neuroscientific inquiries.
VISION successfully predicts human hemodynamic responses as fMRI voxel values to visual inputs with an accuracy exceeding state-of-the-art performance by 45%.
arXiv Detail & Related papers (2023-09-26T15:38:26Z) - BI AVAN: Brain inspired Adversarial Visual Attention Network [67.05560966998559]
We propose a brain-inspired adversarial visual attention network (BI-AVAN) to characterize human visual attention directly from functional brain activity.
Our model imitates the biased competition process between attention-related/neglected objects to identify and locate the visual objects in a movie frame the human brain focuses on in an unsupervised manner.
arXiv Detail & Related papers (2022-10-27T22:20:36Z) - A Symbolic Representation of Human Posture for Interpretable Learning
and Reasoning [2.678461526933908]
We introduce a qualitative spatial reasoning approach that describes the human posture in terms that are more familiar to people.
This paper explores the derivation of our symbolic representation at two levels of detail and its preliminary use as features for interpretable activity recognition.
arXiv Detail & Related papers (2022-10-17T12:22:13Z) - Searching for the Essence of Adversarial Perturbations [73.96215665913797]
We show that adversarial perturbations contain human-recognizable information, which is the key conspirator responsible for a neural network's erroneous prediction.
This concept of human-recognizable information allows us to explain key features related to adversarial perturbations.
arXiv Detail & Related papers (2022-05-30T18:04:57Z) - GIMO: Gaze-Informed Human Motion Prediction in Context [75.52839760700833]
We propose a large-scale human motion dataset that delivers high-quality body pose sequences, scene scans, and ego-centric views with eye gaze.
Our data collection is not tied to specific scenes, which further boosts the motion dynamics observed from our subjects.
To realize the full potential of gaze, we propose a novel network architecture that enables bidirectional communication between the gaze and motion branches.
arXiv Detail & Related papers (2022-04-20T13:17:39Z) - The world seems different in a social context: a neural network analysis
of human experimental data [57.729312306803955]
We show that it is possible to replicate human behavioral data in both individual and social task settings by modifying the precision of prior and sensory signals.
An analysis of the neural activation traces of the trained networks provides evidence that information is coded in fundamentally different ways in the network in the individual and in the social conditions.
arXiv Detail & Related papers (2022-03-03T17:19:12Z) - Leveraging Human Selective Attention for Medical Image Analysis with
Limited Training Data [72.1187887376849]
The selective attention mechanism helps the cognition system focus on task-relevant visual clues by ignoring the presence of distractors.
We propose a framework to leverage gaze for medical image analysis tasks with small training data.
Our method is demonstrated to achieve superior performance on both 3D tumor segmentation and 2D chest X-ray classification tasks.
arXiv Detail & Related papers (2021-12-02T07:55:25Z) - Human brain activity for machine attention [8.673635963837532]
We are the first to exploit neuroscientific data, namely electroencephalography (EEG), to inform a neural attention model about language processing of the human brain.
We devise a method for finding such EEG features to supervise machine attention through combining theoretically motivated cropping with random forest tree splits.
We apply these features to regularise attention on relation classification and show that EEG is more informative than strong baselines.
arXiv Detail & Related papers (2020-06-09T08:39:07Z) - Learning Intermediate Features of Object Affordances with a
Convolutional Neural Network [1.52292571922932]
We train a deep convolutional neural network (CNN) to recognize affordances from images and to learn the underlying features or the dimensionality of affordances.
We view this representational analysis as the first step towards a more formal account of how humans perceive and interact with the environment.
arXiv Detail & Related papers (2020-02-20T19:04:40Z) - Continuous Emotion Recognition via Deep Convolutional Autoencoder and
Support Vector Regressor [70.2226417364135]
It is crucial that the machine should be able to recognize the emotional state of the user with high accuracy.
Deep neural networks have been used with great success in recognizing emotions.
We present a new model for continuous emotion recognition based on facial expression recognition.
arXiv Detail & Related papers (2020-01-31T17:47:16Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.