Attention cannot be an Explanation

• URL: http://arxiv.org/abs/2201.11194v1
• Date: Wed, 26 Jan 2022 21:34:05 GMT
• Title: Attention cannot be an Explanation
• Authors: Arjun R Akula, Song-Chun Zhu
• Abstract summary: We ask how effective are attention based explanations in increasing human trust and reliance in the underlying models? We perform extensive human study experiments that aim to qualitatively and quantitatively assess the degree to which attention based explanations are suitable. Our experiment results show that attention cannot be used as an explanation.
• Score: 99.37090317971312
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Attention based explanations (viz. saliency maps), by providing interpretability to black box models such as deep neural networks, are assumed to improve human trust and reliance in the underlying models. Recently, it has been shown that attention weights are frequently uncorrelated with gradient-based measures of feature importance. Motivated by this, we ask a follow-up question: "Assuming that we only consider the tasks where attention weights correlate well with feature importance, how effective are these attention based explanations in increasing human trust and reliance in the underlying models?". In other words, can we use attention as an explanation? We perform extensive human study experiments that aim to qualitatively and quantitatively assess the degree to which attention based explanations are suitable in increasing human trust and reliance. Our experiment results show that attention cannot be used as an explanation.

Related papers

• Guiding Visual Question Answering with Attention Priors [76.21671164766073]
  We propose to guide the attention mechanism using explicit linguistic-visual grounding. This grounding is derived by connecting structured linguistic concepts in the query to their referents among the visual objects. The resultant algorithm is capable of probing attention-based reasoning models, injecting relevant associative knowledge, and regulating the core reasoning process.
  arXiv  Detail & Related papers  (2022-05-25T09:53:47Z)
• Do Transformer Models Show Similar Attention Patterns to Task-Specific Human Gaze? [0.0]
  Self-attention functions in state-of-the-art NLP models often correlate with human attention. We investigate whether self-attention in large-scale pre-trained language models is as predictive of human eye fixation patterns during task-reading as classical cognitive models of human attention.
  arXiv  Detail & Related papers  (2022-04-25T08:23:13Z)
• Attention in Reasoning: Dataset, Analysis, and Modeling [31.3104693230952]
  We propose an Attention with Reasoning capability (AiR) framework that uses attention to understand and improve the process leading to task outcomes. We first define an evaluation metric based on a sequence of atomic reasoning operations, enabling a quantitative measurement of attention. We then collect human eye-tracking and answer correctness data, and analyze various machine and human attention mechanisms on their reasoning capability.
  arXiv  Detail & Related papers  (2022-04-20T20:32:31Z)
• Rethinking Attention-Model Explainability through Faithfulness Violation Test [29.982295060192904]
  We study the explainability of current attention-based techniques, such as Attentio$odot$Gradient and LRP-based attention explanations. We show that most tested explanation methods are unexpectedly hindered by the faithfulness violation issue.
  arXiv  Detail & Related papers  (2022-01-28T13:42:31Z)
• Counterfactual Attention Learning for Fine-Grained Visual Categorization and Re-identification [101.49122450005869]
  We present a counterfactual attention learning method to learn more effective attention based on causal inference. Specifically, we analyze the effect of the learned visual attention on network prediction. We evaluate our method on a wide range of fine-grained recognition tasks.
  arXiv  Detail & Related papers  (2021-08-19T14:53:40Z)
• Is Sparse Attention more Interpretable? [52.85910570651047]
  We investigate how sparsity affects our ability to use attention as an explainability tool. We find that only a weak relationship between inputs and co-indexed intermediate representations exists -- under sparse attention. We observe in this setting that inducing sparsity may make it less plausible that attention can be used as a tool for understanding model behavior.
  arXiv  Detail & Related papers  (2021-06-02T11:42:56Z)
• Why Attentions May Not Be Interpretable? [46.69116768203185]
  Recent research found that attention-as-importance interpretations often do not work as we expected. We show that one root cause of this phenomenon is shortcuts, which means that the attention weights themselves may carry extra information. We propose two methods to mitigate this issue.
  arXiv  Detail & Related papers  (2020-06-10T05:08:30Z)
• Exploring Self-attention for Image Recognition [151.12000247183636]
  We consider two forms of self-attention for image recognition. One is pairwise self-attention, which generalizes standard dot-product attention. The other is patchwise self-attention, which is strictly more powerful than convolution.
  arXiv  Detail & Related papers  (2020-04-28T16:01:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.