Interpretability-Guided Soft Pruning of Attention Heads in Vision Transformers

• URL: http://arxiv.org/abs/2608.00264v1
• Date: Fri, 31 Jul 2026 20:14:03 GMT
• Title: Interpretability-Guided Soft Pruning of Attention Heads in Vision Transformers
• Abstract summary: We propose a spectral analysis and new visualization technique for individual attention heads based on the Laplacian eigenvectors of their attention maps.<n>Building upon recent observations regarding the block structure of Vision Transformers, we perform semantic clustering of attention heads and identify functional redundancies.<n>We introduce SAPER, an end-to-end differentiable pruning framework based on the LapSum Soft Top-K approach.
• Score: 8.49520972146025
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision foundation models, such as DINOv2, learn highly expressive representations but rely on massive, opaque architectures that demand substantial computational power and memory. To provide an interpretable-guided and efficient solution to this issue, we first propose a spectral analysis and new visualization technique for individual attention heads based on the Laplacian eigenvectors of their attention maps. Building upon recent observations regarding the block structure of Vision Transformers, we perform semantic clustering of attention heads and identify functional redundancies. Leveraging these insights, we introduce SAPER (Soft Attention PrunER), an end-to-end differentiable pruning framework based on the LapSum Soft Top-K approach. Extensive experiments on ImageNet-1K demonstrate that SAPER achieves a highly favorable accuracy-efficiency trade-off, outperforming the competitive RAPTOR baseline in FLOPs reduction while preserving strong classification performance.

Related papers

• VEN-VL: A Visual Ensemble MoE Framework for Effective and Efficient Multi-Modal Understanding [50.904820090651974]
  VEN-VL is a visual ensemble MoE framework for effective and efficient perception.<n>We first enrich the information capacity by unifying the visual representations of different perspectives.<n>We then progressively compact it with adaptive routers in specialized visual experts to enhance the information density.
  arXiv  Detail & Related papers  (2026-05-25T15:28:48Z)
• Accelerating Vision Foundation Models with Drop-in Depthwise Convolution [51.50107862675191]
  We introduce an efficient depthwise convolution-based layer that serves as a drop-in replacement for attention heads.<n>Across both image classification and segmentation tasks, our method achieves 17-20% percent inference speedup with minimal performance degradation.
  arXiv  Detail & Related papers  (2026-05-21T08:07:23Z)
• Large Vision-Language Models Get Lost in Attention [51.851592109135716]
  We propose a unified framework grounded in information theory and geometry to quantify the geometric and entropic nature of residual updates.<n>Applying this unified framework reveals a fundamental functional decoupling: Attention acts as a subspace-language operator focused on reconfiguration, whereas FFNs serve as subspace-language operators driving semantic innovation.
  arXiv  Detail & Related papers  (2026-05-07T04:45:52Z)
• Benchmarking Vision Foundation Models for Domain-Generalizable Face Anti-Spoofing [2.099922236065961]
  Face Anti-Spoofing (FAS) remains challenging due to the requirement for robust domain generalization across unseen environments.<n>This paper revisits the potential of vision-only foundation models to establish a highly efficient and robust baseline for FAS.<n>Our comprehensive analysis reveals that self-supervised vision models, particularly DINOv2 with Registers, significantly suppress attention artifacts and capture critical, fine-grained spoofing cues.
  arXiv  Detail & Related papers  (2026-04-21T08:05:21Z)
• StepVAR: Structure-Texture Guided Pruning for Visual Autoregressive Models [98.72926158261937]
  We propose a training-free token pruning framework for Visual AutoRegressive models.<n>We employ a lightweight high-pass filter to capture local texture details, while leveraging Principal Component Analysis (PCA) to preserve global structural information.<n>To maintain valid next-scale prediction under sparse tokens, we introduce a nearest neighbor feature propagation strategy.
  arXiv  Detail & Related papers  (2026-03-02T11:35:05Z)
• CAPA: Contribution-Aware Pruning and FFN Approximation for Efficient Large Vision-Language Models [14.30682201364961]
  We show that Attention Contribution, which weights attention probabilities by value vector magnitude, provides a more accurate criterion for visual token selection.<n>We introduce CAPA (Contribution-Aware Pruning and FFN Approximation), a dual-strategy framework that prunes visual tokens using attention contribution at critical functional transitions.
  arXiv  Detail & Related papers  (2026-01-30T19:09:03Z)
• The Loupe: A Plug-and-Play Attention Module for Amplifying Discriminative Features in Vision Transformers [0.0]
  We introduce The Loupe, a lightweight, plug-and-play attention module designed to be inserted into pre-trained backbones like the Swin Transformer.<n>The Loupe is trained end-to-end with a composite loss function that implicitly guides the model to focus on the most discriminative object parts.<n>Our experimental evaluation on the challenging CUB-200-2011 dataset shows that The Loupe improves the accuracy of a Swin-Base model from 85.40% to 88.06%, a significant gain of 2.66%.
  arXiv  Detail & Related papers  (2025-08-20T19:07:21Z)
• Top-Down Compression: Revisit Efficient Vision Token Projection for Visual Instruction Tuning [70.57180215148125]
  Visual instruction tuning aims to enable large language models to comprehend the visual world.<n>Existing methods often grapple with the intractable trade-off between accuracy and efficiency.<n>We present LLaVA-Meteor, a novel approach that strategically compresses visual tokens without compromising core information.
  arXiv  Detail & Related papers  (2025-05-17T10:22:29Z)
• GMAR: Gradient-Driven Multi-Head Attention Rollout for Vision Transformer Interpretability [0.20482269513546453]
  The Vision Transformer (ViT) has made significant advancements in computer vision, utilizing self-attention mechanisms to achieve state-of-the-art performance across various tasks.<n>The intricate multi-head attention mechanism of ViT presents significant challenges to interpretability, as the underlying prediction process remains opaque.<n>We introduce Gradient-Driven Multi-Head Attention Rollout (GMAR), a novel method that quantifies the importance of each attention head using gradient-based scores.
  arXiv  Detail & Related papers  (2025-04-28T01:58:39Z)
• "Principal Components" Enable A New Language of Images [79.45806370905775]
  We introduce a novel visual tokenization framework that embeds a provable PCA-like structure into the latent token space.<n>Our approach achieves state-of-the-art reconstruction performance and enables better interpretability to align with the human vision system.
  arXiv  Detail & Related papers  (2025-03-11T17:59:41Z)
• Differential Contrastive Training for Gaze Estimation [24.53837441433775]
  We propose a novel Differential Contrastive Training strategy, which boosts gaze estimation performance with the help of the CLIP.<n>A Differential Contrastive Gaze Estimation network (DCGaze) composed of a Visual Appearance-aware branch and a Semantic Differential-aware branch is introduced.
  arXiv  Detail & Related papers  (2025-02-27T14:23:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.