Transformer Interpretability from Perspective of Attention and Gradient

• URL: http://arxiv.org/abs/2605.11392v1
• Date: Tue, 12 May 2026 01:28:01 GMT
• Title: Transformer Interpretability from Perspective of Attention and Gradient
• Abstract summary: Gradient is widely utilized in Transformer interpretation.<n>We propose a method to achieve it by guiding the gradient direction, or more precisely, the attention direction.
• Score: 47.816447130827555
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Although researchers' attention is more focused on the performance of Transformer models, the interpretation of Transformer can never be ignored. Gradient is widely utilized in Transformer interpretation. From the perspective of attention and gradient, we conduct an in-depth study of Transformer interpretation and propose a method to achieve it by guiding the gradient direction, or more precisely, the attention direction. The method enables more comprehensive interpretation of feature regions, offers detail interpretation, and helps to better understand Transformer mechanism. Leveraging the difference in how Vision Transformer (ViT) and humans perceive images, we alter the class of an image in a way that is almost imperceptible to the human eye. This class rewriting phenomenon may potentially pose security risks in certain scenarios.

Related papers

• Disentangling Visual Transformers: Patch-level Interpretability for Image Classification [2.899118947717404]
  We propose Hindered Transformer (HiT), a novel interpretable by design architecture inspired by visual transformers.<n>HiT can be interpreted as a linear combination of patch-level information.<n>We show that the advantages of our approach in terms of explicability come with a reasonable trade-off in performance.
  arXiv  Detail & Related papers  (2025-02-24T14:30:29Z)
• Inspecting Explainability of Transformer Models with Additional Statistical Information [24.064325847615546]
  Chefer et al. can visualize the Transformer on vision and multi-modal tasks effectively by combining attention layers to show the importance of each image patch.<n>However, when applying to other variants of Transformer such as the Swin Transformer, this method can not focus on the predicted object.<n>Our method, by considering the statistics of tokens in layer normalization layers, shows a great ability to interpret the explainability of Swin Transformer and ViT.
  arXiv  Detail & Related papers  (2023-11-19T17:22:50Z)
• Vision Transformers for Computer Go [4.111084095218968]
  This study focuses on the analysis of the Transformer in Vision. Through a detailed analysis of numerous points such as prediction accuracy, win rates, memory, speed, size, or even learning rate, we have been able to highlight the substantial role that transformers can play in the game of Go.
  arXiv  Detail & Related papers  (2023-09-22T07:35:37Z)
• Image Deblurring by Exploring In-depth Properties of Transformer [86.7039249037193]
  We leverage deep features extracted from a pretrained vision transformer (ViT) to encourage recovered images to be sharp without sacrificing the performance measured by the quantitative metrics. By comparing the transformer features between recovered image and target one, the pretrained transformer provides high-resolution blur-sensitive semantic information. One regards the features as vectors and computes the discrepancy between representations extracted from recovered image and target one in Euclidean space.
  arXiv  Detail & Related papers  (2023-03-24T14:14:25Z)
• Are More Layers Beneficial to Graph Transformers? [97.05661983225603]
  Current graph transformers suffer from the bottleneck of improving performance by increasing depth. Deep graph transformers are limited by the vanishing capacity of global attention. We propose a novel graph transformer model named DeepGraph that explicitly employs substructure tokens in the encoded representation.
  arXiv  Detail & Related papers  (2023-03-01T15:22:40Z)
• A Light Touch Approach to Teaching Transformers Multi-view Geometry [80.35521056416242]
  We propose a "light touch" approach to guiding visual Transformers to learn multiple-view geometry. We achieve this by using epipolar lines to guide the Transformer's cross-attention maps. Unlike previous methods, our proposal does not require any camera pose information at test-time.
  arXiv  Detail & Related papers  (2022-11-28T07:54:06Z)
• XAI for Transformers: Better Explanations through Conservative Propagation [60.67748036747221]
  We show that the gradient in a Transformer reflects the function only locally, and thus fails to reliably identify the contribution of input features to the prediction. Our proposal can be seen as a proper extension of the well-established LRP method to Transformers.
  arXiv  Detail & Related papers  (2022-02-15T10:47:11Z)
• Transformer visualization via dictionary learning: contextualized embedding as a linear superposition of transformer factors [15.348047288817478]
  We propose to use dictionary learning to open up "black boxes" as linear superpositions of transformer factors. Through visualization, we demonstrate the hierarchical semantic structures captured by the transformer factors. We hope this visualization tool can bring further knowledge and a better understanding of how transformer networks work.
  arXiv  Detail & Related papers  (2021-03-29T20:51:33Z)
• A Survey on Visual Transformer [126.56860258176324]
  Transformer is a type of deep neural network mainly based on the self-attention mechanism. In this paper, we review these vision transformer models by categorizing them in different tasks and analyzing their advantages and disadvantages.
  arXiv  Detail & Related papers  (2020-12-23T09:37:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.