Gradient-Skipping Relevance Propagation for Efficient Explainability of Vision Transformers

• URL: http://arxiv.org/abs/2607.10365v1
• Date: Sat, 11 Jul 2026 15:43:09 GMT
• Title: Gradient-Skipping Relevance Propagation for Efficient Explainability of Vision Transformers
• Abstract summary: We introduce GradSkip, a novel relevance propagation method for ViTs based on adaptive head weighting and skip-aware propagation.<n>Experiments on ImageNet1K and BloodMNIST demonstrate GradSkip's faithfulness while requiring over 14 times fewer GFLOPs than the best-performing existing approaches.
• Score: 5.770347328961063
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision Transformers (ViTs) are difficult to interpret because current methods of relevance propagation and attention flow do not fully consider some key architectural features, such as the uneven importance of attention heads and residual connections. Prior approaches typically assume uniform importance across attention heads; furthermore, they model skip connections as identity paths, leading to inaccurate relevance attribution. To address these issues, we introduce GradSkip, a novel relevance propagation method for ViTs based on adaptive head weighting and skip-aware propagation. GradSkip models the different importance of the attention heads and dynamically distributes relevance between the attention and residual paths. Experiments on ImageNet1K and BloodMNIST demonstrate a state-of-the-art faithfulness of GradSkip while requiring over 14 times fewer GFLOPs than the best-performing existing approaches. Additional evaluations using transformer-based segmentation confirm improved localization and alignment with ground-truth regions.

Related papers

• HyTBE: Hyperbolic Target-Background Expert Model for Cross-Domain Infrared Small Target Detection [3.6870604311531885]
  Infrared small target detection (IRSTD) has achieved substantial progress under domain-consistent evaluation.<n>We propose HyTBE, a Hyperbolic Target-Background Expert model that expands source-domain relation patterns.<n>Experiments on NUAA-SIRST, NUDT-SIRST, and IRSTD-1K demonstrate that HyTBE achieves stronger cross-domain generalization than competitive baselines.
  arXiv  Detail & Related papers  (2026-08-06T09:02:09Z)
• Large Vision-Language Models Get Lost in Attention [51.851592109135716]
  We propose a unified framework grounded in information theory and geometry to quantify the geometric and entropic nature of residual updates.<n>Applying this unified framework reveals a fundamental functional decoupling: Attention acts as a subspace-language operator focused on reconfiguration, whereas FFNs serve as subspace-language operators driving semantic innovation.
  arXiv  Detail & Related papers  (2026-05-07T04:45:52Z)
• Decision-Aware Attention Propagation for Vision Transformer Explainability [1.9116784879310027]
  Decision-Aware Attention Propagation (DAP) is an attribution method that injects decision-relevant priors into transformer attention propagation.<n>DAP produces attribution maps that are more class sensitive, compact, and faithful than those generated by conventional attention based methods.
  arXiv  Detail & Related papers  (2026-04-20T11:10:18Z)
• Learn to Rank: Visual Attribution by Learning Importance Ranking [58.69028273772474]
  We propose a new approach to generate visual attribution maps for computer vision models.<n>Our method produces dense, pixel-level attributions in a single forward pass with optional, few-step gradient refinement.<n>Our experiments demonstrate consistent quantitative improvements and sharper, boundary-aligned explanations.
  arXiv  Detail & Related papers  (2026-04-07T12:53:22Z)
• Multi-Cue Adaptive Visual Token Pruning for Large Vision-Language Models [85.51753014478315]
  We introduce AdaptPrune, a novel plug-and-play training-free pruning method.<n>It builds on conventional attention-based pruning by integrating spatial distance and token similarity with an adaptive NMS approach.<n>Our approach ensures a comprehensive evaluation of token importance and substantially refines the pruning decisions.
  arXiv  Detail & Related papers  (2025-03-11T03:58:17Z)
• Scaled and Inter-token Relation Enhanced Transformer for Sample-restricted Residential NILM [0.0]
  We propose a novel transformer architecture with two key innovations: inter-token relation enhancement and dynamic temperature tuning.<n>We validate our method on the REDD dataset and show that it outperforms the original transformer and state-of-the-art models by 10-15% in F1 score across various appliance types.
  arXiv  Detail & Related papers  (2024-10-12T18:58:45Z)
• MimiQ: Low-Bit Data-Free Quantization of Vision Transformers with Encouraging Inter-Head Attention Similarity [32.532780329341186]
  Data-free quantization (DFQ) is a technique that creates a lightweight network from its full-precision counterpart without the original training data, often through a synthetic dataset.<n>Several DFQ methods have been proposed for vision transformer (ViT) architectures, but they fail to achieve efficacy in low-bit settings.<n>We devise MimiQ, a novel DFQ method designed for ViTs that enhances inter-head attention similarity.
  arXiv  Detail & Related papers  (2024-07-29T13:57:40Z)
• DAT++: Spatially Dynamic Vision Transformer with Deformable Attention [87.41016963608067]
  We present Deformable Attention Transformer ( DAT++), a vision backbone efficient and effective for visual recognition. DAT++ achieves state-of-the-art results on various visual recognition benchmarks, with 85.9% ImageNet accuracy, 54.5 and 47.0 MS-COCO instance segmentation mAP, and 51.5 ADE20K semantic segmentation mIoU.
  arXiv  Detail & Related papers  (2023-09-04T08:26:47Z)
• Generic Attention-model Explainability by Weighted Relevance Accumulation [9.816810016935541]
  We propose a weighted relevancy strategy, which takes the importance of token values into consideration, to reduce distortion when equally accumulating relevance. To evaluate our method, we propose a unified CLIP-based two-stage model, named CLIPmapper, to process Vision-and-Language tasks.
  arXiv  Detail & Related papers  (2023-08-20T12:02:30Z)
• Calibrating Undisciplined Over-Smoothing in Transformer for Weakly Supervised Semantic Segmentation [51.14107156747967]
  Weakly supervised semantic segmentation (WSSS) has attracted considerable attention because it requires fewer annotations than fully supervised approaches.<n>We propose an Adaptive Re-Activation Mechanism (AReAM) to control deep-level attention to undisciplined over-smoothing.<n>AReAM substantially improves segmentation performance compared with existing WSSS methods, reducing noise while sharpening focus on relevant semantic regions.
  arXiv  Detail & Related papers  (2023-05-04T19:11:33Z)
• From Saliency to DINO: Saliency-guided Vision Transformer for Few-shot Keypoint Detection [36.9781808268263]
  Few-shot keypoint detection (FSKD) attempts to localize any keypoints, including novel or base keypoints, depending on the reference samples. FSKD requires the semantically meaningful relations for keypoint similarity learning to overcome the ubiquitous noise and ambiguous local patterns. We present a novel saliency-guided vision transformer, dubbed SalViT, for few-shot keypoint detection.
  arXiv  Detail & Related papers  (2023-04-06T15:22:34Z)
• Transformer Interpretability Beyond Attention Visualization [87.96102461221415]
  Self-attention techniques, and specifically Transformers, are dominating the field of text processing. In this work, we propose a novel way to compute relevancy for Transformer networks.
  arXiv  Detail & Related papers  (2020-12-17T18:56:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.