Rethinking Attention-Model Explainability through Faithfulness Violation Test

• URL: http://arxiv.org/abs/2201.12114v1
• Date: Fri, 28 Jan 2022 13:42:31 GMT
• Title: Rethinking Attention-Model Explainability through Faithfulness Violation Test
• Authors: Yibing Liu, Haoliang Li, Yangyang Guo, Chenqi Kong, Jing Li, Shiqi Wang
• Abstract summary: We study the explainability of current attention-based techniques, such as Attentio$odot$Gradient and LRP-based attention explanations. We show that most tested explanation methods are unexpectedly hindered by the faithfulness violation issue.
• Score: 29.982295060192904
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Attention mechanisms are dominating the explainability of deep models. They produce probability distributions over the input, which are widely deemed as feature-importance indicators. However, in this paper, we find one critical limitation in attention explanations: weakness in identifying the polarity of feature impact. This would be somehow misleading -- features with higher attention weights may not faithfully contribute to model predictions; instead, they can impose suppression effects. With this finding, we reflect on the explainability of current attention-based techniques, such as Attentio$\odot$Gradient and LRP-based attention explanations. We first propose an actionable diagnostic methodology (henceforth faithfulness violation test) to measure the consistency between explanation weights and the impact polarity. Through the extensive experiments, we then show that most tested explanation methods are unexpectedly hindered by the faithfulness violation issue, especially the raw attention. Empirical analyses on the factors affecting violation issues further provide useful observations for adopting explanation methods in attention models.

Related papers

• Investigating the Impact of Model Instability on Explanations and Uncertainty [43.254616360807496]
  We simulate uncertainty in text input by introducing noise at inference time. We find that high uncertainty doesn't necessarily imply low explanation plausibility. This suggests that noise-augmented models may be better at identifying salient tokens when uncertain.
  arXiv  Detail & Related papers  (2024-02-20T13:41:21Z)
• Revisiting the Fragility of Influence Functions [1.4699455652461724]
  influence functions, which approximates the effect that leave-one-out has on loss, has been proposed to verify accuracy or faithfulness of deep learning models. Here, we analyze key metrics that are used to validate influence functions. Our results indicate that the procedures may cause their validation work to be unclear.
  arXiv  Detail & Related papers  (2023-03-22T21:36:56Z)
• Guiding Visual Question Answering with Attention Priors [76.21671164766073]
  We propose to guide the attention mechanism using explicit linguistic-visual grounding. This grounding is derived by connecting structured linguistic concepts in the query to their referents among the visual objects. The resultant algorithm is capable of probing attention-based reasoning models, injecting relevant associative knowledge, and regulating the core reasoning process.
  arXiv  Detail & Related papers  (2022-05-25T09:53:47Z)
• Attention cannot be an Explanation [99.37090317971312]
  We ask how effective are attention based explanations in increasing human trust and reliance in the underlying models? We perform extensive human study experiments that aim to qualitatively and quantitatively assess the degree to which attention based explanations are suitable. Our experiment results show that attention cannot be used as an explanation.
  arXiv  Detail & Related papers  (2022-01-26T21:34:05Z)
• Is Sparse Attention more Interpretable? [52.85910570651047]
  We investigate how sparsity affects our ability to use attention as an explainability tool. We find that only a weak relationship between inputs and co-indexed intermediate representations exists -- under sparse attention. We observe in this setting that inducing sparsity may make it less plausible that attention can be used as a tool for understanding model behavior.
  arXiv  Detail & Related papers  (2021-06-02T11:42:56Z)
• Why Attentions May Not Be Interpretable? [46.69116768203185]
  Recent research found that attention-as-importance interpretations often do not work as we expected. We show that one root cause of this phenomenon is shortcuts, which means that the attention weights themselves may carry extra information. We propose two methods to mitigate this issue.
  arXiv  Detail & Related papers  (2020-06-10T05:08:30Z)
• Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals [53.484562601127195]
  We point out the inability to infer behavioral conclusions from probing results. We offer an alternative method that focuses on how the information is being used, rather than on what information is encoded.
  arXiv  Detail & Related papers  (2020-06-01T15:00:11Z)
• Towards Transparent and Explainable Attention Models [34.0557018891191]
  We first explain why current attention mechanisms in LSTM based encoders can neither provide a faithful nor a plausible explanation of the model's predictions. We propose a modified LSTM cell with a diversity-driven training objective that ensures that the hidden representations learned at different time steps are diverse. Human evaluations indicate that the attention distributions learned by our model offer a plausible explanation of the model's predictions.
  arXiv  Detail & Related papers  (2020-04-29T14:47:50Z)
• Explain and Improve: LRP-Inference Fine-Tuning for Image Captioning Models [82.3793660091354]
  This paper analyzes the predictions of image captioning models with attention mechanisms beyond visualizing the attention itself. We develop variants of layer-wise relevance propagation (LRP) and gradient-based explanation methods, tailored to image captioning models with attention mechanisms.
  arXiv  Detail & Related papers  (2020-01-04T05:15:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.