Regularization, Semi-supervision, and Supervision for a Plausible Attention-Based Explanation

• URL: http://arxiv.org/abs/2501.12775v1
• Date: Wed, 22 Jan 2025 10:17:20 GMT
• Title: Regularization, Semi-supervision, and Supervision for a Plausible Attention-Based Explanation
• Authors: Duc Hau Nguyen, Cyrielle Mallart, Guillaume Gravier, Pascale Sébillot,
• Abstract summary: Empirical studies postulate that attention maps can be provided as an explanation for model output. Recent studies show that attention weights in the RNN encoders are hardly plausible because they spread on input tokens. We propose 3 additional constraints to the learning objective function to improve the plausibility of the attention map.
• Score: 0.2499907423888049
• License:
• Abstract: Attention mechanism is contributing to the majority of recent advances in machine learning for natural language processing. Additionally, it results in an attention map that shows the proportional influence of each input in its decision. Empirical studies postulate that attention maps can be provided as an explanation for model output. However, it is still questionable to ask whether this explanation helps regular people to understand and accept the model output (the plausibility of the explanation). Recent studies show that attention weights in the RNN encoders are hardly plausible because they spread on input tokens. We thus propose 3 additional constraints to the learning objective function to improve the plausibility of the attention map: regularization to increase the attention weight sparsity, semi-supervision to supervise the map by a heuristic and supervision by human annotation. Results show that all techniques can improve the attention map plausibility at some level. We also observe that specific instructions for human annotation might have a negative effect on classification performance. Beyond the attention map, the result of experiments on text classification tasks also shows that no matter how the constraint brings the gain, the contextualization layer plays a crucial role in finding the right space for finding plausible tokens.

Related papers

• Reversed Attention: On The Gradient Descent Of Attention Layers In GPT [55.2480439325792]
  We study the mathematics of the backward pass of attention, revealing that it implicitly calculates an attention matrix we refer to as "Reversed Attention" In an experimental setup, we showcase the ability of Reversed Attention to directly alter the forward pass of attention, without modifying the model's weights. In addition to enhancing the comprehension of how LM configure attention layers during backpropagation, Reversed Attention maps contribute to a more interpretable backward pass.
  arXiv  Detail & Related papers  (2024-12-22T13:48:04Z)
• Perturbation-based Self-supervised Attention for Attention Bias in Text Classification [31.144857032681905]
  We propose a perturbation-based self-supervised attention approach to guide attention learning. We add as much noise as possible to all the words in the sentence without changing their semantics and predictions. Experimental results on three text classification tasks show that our approach can significantly improve the performance of current attention-based models.
  arXiv  Detail & Related papers  (2023-05-25T03:18:18Z)
• Guiding Visual Question Answering with Attention Priors [76.21671164766073]
  We propose to guide the attention mechanism using explicit linguistic-visual grounding. This grounding is derived by connecting structured linguistic concepts in the query to their referents among the visual objects. The resultant algorithm is capable of probing attention-based reasoning models, injecting relevant associative knowledge, and regulating the core reasoning process.
  arXiv  Detail & Related papers  (2022-05-25T09:53:47Z)
• Rethinking Attention-Model Explainability through Faithfulness Violation Test [29.982295060192904]
  We study the explainability of current attention-based techniques, such as Attentio$odot$Gradient and LRP-based attention explanations. We show that most tested explanation methods are unexpectedly hindered by the faithfulness violation issue.
  arXiv  Detail & Related papers  (2022-01-28T13:42:31Z)
• Attention cannot be an Explanation [99.37090317971312]
  We ask how effective are attention based explanations in increasing human trust and reliance in the underlying models? We perform extensive human study experiments that aim to qualitatively and quantitatively assess the degree to which attention based explanations are suitable. Our experiment results show that attention cannot be used as an explanation.
  arXiv  Detail & Related papers  (2022-01-26T21:34:05Z)
• Is Sparse Attention more Interpretable? [52.85910570651047]
  We investigate how sparsity affects our ability to use attention as an explainability tool. We find that only a weak relationship between inputs and co-indexed intermediate representations exists -- under sparse attention. We observe in this setting that inducing sparsity may make it less plausible that attention can be used as a tool for understanding model behavior.
  arXiv  Detail & Related papers  (2021-06-02T11:42:56Z)
• SparseBERT: Rethinking the Importance Analysis in Self-attention [107.68072039537311]
  Transformer-based models are popular for natural language processing (NLP) tasks due to its powerful capacity. Attention map visualization of a pre-trained model is one direct method for understanding self-attention mechanism. We propose a Differentiable Attention Mask (DAM) algorithm, which can be also applied in guidance of SparseBERT design.
  arXiv  Detail & Related papers  (2021-02-25T14:13:44Z)
• Why Attentions May Not Be Interpretable? [46.69116768203185]
  Recent research found that attention-as-importance interpretations often do not work as we expected. We show that one root cause of this phenomenon is shortcuts, which means that the attention weights themselves may carry extra information. We propose two methods to mitigate this issue.
  arXiv  Detail & Related papers  (2020-06-10T05:08:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.