Staying True to Your Word: (How) Can Attention Become Explanation?

• URL: http://arxiv.org/abs/2005.09379v1
• Date: Tue, 19 May 2020 11:55:11 GMT
• Title: Staying True to Your Word: (How) Can Attention Become Explanation?
• Authors: Martin Tutek, Jan \v{S}najder
• Abstract summary: We provide an explanation as to why attention has seen rightful critique when used with recurrent networks in sequence classification tasks. We propose a remedy to these issues in the form of a word level objective.
• Score: 0.17767466724342063
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The attention mechanism has quickly become ubiquitous in NLP. In addition to improving performance of models, attention has been widely used as a glimpse into the inner workings of NLP models. The latter aspect has in the recent years become a common topic of discussion, most notably in work of Jain and Wallace, 2019; Wiegreffe and Pinter, 2019. With the shortcomings of using attention weights as a tool of transparency revealed, the attention mechanism has been stuck in a limbo without concrete proof when and whether it can be used as an explanation. In this paper, we provide an explanation as to why attention has seen rightful critique when used with recurrent networks in sequence classification tasks. We propose a remedy to these issues in the form of a word level objective and our findings give credibility for attention to provide faithful interpretations of recurrent models.

Related papers

• Reversed Attention: On The Gradient Descent Of Attention Layers In GPT [55.2480439325792]
  We study the mathematics of the backward pass of attention, revealing that it implicitly calculates an attention matrix we refer to as "Reversed Attention" In an experimental setup, we showcase the ability of Reversed Attention to directly alter the forward pass of attention, without modifying the model's weights. In addition to enhancing the comprehension of how LM configure attention layers during backpropagation, Reversed Attention maps contribute to a more interpretable backward pass.
  arXiv  Detail & Related papers  (2024-12-22T13:48:04Z)
• Guiding Visual Question Answering with Attention Priors [76.21671164766073]
  We propose to guide the attention mechanism using explicit linguistic-visual grounding. This grounding is derived by connecting structured linguistic concepts in the query to their referents among the visual objects. The resultant algorithm is capable of probing attention-based reasoning models, injecting relevant associative knowledge, and regulating the core reasoning process.
  arXiv  Detail & Related papers  (2022-05-25T09:53:47Z)
• Attention cannot be an Explanation [99.37090317971312]
  We ask how effective are attention based explanations in increasing human trust and reliance in the underlying models? We perform extensive human study experiments that aim to qualitatively and quantitatively assess the degree to which attention based explanations are suitable. Our experiment results show that attention cannot be used as an explanation.
  arXiv  Detail & Related papers  (2022-01-26T21:34:05Z)
• Learning to ignore: rethinking attention in CNNs [87.01305532842878]
  We propose to reformulate the attention mechanism in CNNs to learn to ignore instead of learning to attend. Specifically, we propose to explicitly learn irrelevant information in the scene and suppress it in the produced representation.
  arXiv  Detail & Related papers  (2021-11-10T13:47:37Z)
• Is Sparse Attention more Interpretable? [52.85910570651047]
  We investigate how sparsity affects our ability to use attention as an explainability tool. We find that only a weak relationship between inputs and co-indexed intermediate representations exists -- under sparse attention. We observe in this setting that inducing sparsity may make it less plausible that attention can be used as a tool for understanding model behavior.
  arXiv  Detail & Related papers  (2021-06-02T11:42:56Z)
• Improving the Faithfulness of Attention-based Explanations with Task-specific Information for Text Classification [9.147707153504117]
  We propose a new family of Task-Scaling (TaSc) mechanisms that learn task-specific non-contextualised information to scale the original attention weights. TaSc consistently provides more faithful attention-based explanations compared to three widely-used interpretability techniques.
  arXiv  Detail & Related papers  (2021-05-06T13:35:03Z)
• Why Attentions May Not Be Interpretable? [46.69116768203185]
  Recent research found that attention-as-importance interpretations often do not work as we expected. We show that one root cause of this phenomenon is shortcuts, which means that the attention weights themselves may carry extra information. We propose two methods to mitigate this issue.
  arXiv  Detail & Related papers  (2020-06-10T05:08:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.