Rényi Attention Entropy for Patch Pruning

• URL: http://arxiv.org/abs/2604.03803v1
• Date: Sat, 04 Apr 2026 17:10:17 GMT
• Title: Rényi Attention Entropy for Patch Pruning
• Abstract summary: We introduce a criterion based on the Shannon entropy of the attention distribution to identify informative patches for pruning.<n>We also extend the criterion from Shannon to Rényi entropy, which emphasizes sharp attention peaks.<n>In experiments on fine-grained image recognition, where patch selection is critical, our method reduced computation while preserving accuracy.
• Score: 0.42970700836450487
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Transformers are strong baselines in both vision and language because self-attention captures long-range dependencies across tokens. However, the cost of self-attention grows quadratically with the number of tokens. Patch pruning mitigates this cost by estimating per-patch importance and removing redundant patches. To identify informative patches for pruning, we introduce a criterion based on the Shannon entropy of the attention distribution. Low-entropy patches, which receive selective and concentrated attention, are kept as important, while high-entropy patches with attention spread across many locations are treated as redundant. We also extend the criterion from Shannon to Rényi entropy, which emphasizes sharp attention peaks and supports pruning strategies that adapt to task needs and computational limits. In experiments on fine-grained image recognition, where patch selection is critical, our method reduced computation while preserving accuracy. Moreover, adjusting the pruning policy through the Rényi entropy measure yields further gains and improves the trade-off between accuracy and computation.

Related papers

• Evading Visual Aphasia: Contrastive Adaptive Semantic Token Pruning for Vision-Language Models [52.78477729846771]
  We introduce COAST (COntrastive Adaptive Semantic Token Pruning), a training-free pruning framework that casts compression as adaptive semantic routing.<n> COAST consistently outperforms strong pruning baselines across token budgets and generalizes across multiple LVLM families.
  arXiv  Detail & Related papers  (2026-05-10T09:07:04Z)
• Probability-Entropy Calibration: An Elastic Indicator for Adaptive Fine-tuning [55.2818264614932]
  RankTuner introduces a probability--entropy calibration signal, the Relative Rank Indicator, which compares the rank of the ground-truth token with its expected rank under the prediction distribution.<n>The inverse indicator is used as a token-wise Relative Scale to reweight the fine-tuning objective, focusing updates on truly under-learned tokens.
  arXiv  Detail & Related papers  (2026-02-02T07:27:19Z)
• Rectified SpaAttn: Revisiting Attention Sparsity for Efficient Video Generation [22.35209793690791]
  Diffusion Transformers dominate video generation, but the quadratic complexity of attention introduces substantial latency.<n> Attention sparsity reduces computational costs by focusing on critical tokens while ignoring non-critical tokens.<n>Existing methods induce systematic biases in attention allocation.<n>We propose Rectified SpaAttn, which rectifies attention allocation with implicit full attention reference.
  arXiv  Detail & Related papers  (2025-11-25T02:03:54Z)
• DoPE: Denoising Rotary Position Embedding [60.779039511252584]
  Rotary Position Embedding (RoPE) in Transformer models has inherent limits that weaken length.<n>We reinterpret the attention map with positional encoding as a noisy feature map, and propose Denoising Positional extrapolation page (DoPE)<n>DoPE is a training-free method based on truncated matrix entropy to detect outlier frequency bands in the feature map.
  arXiv  Detail & Related papers  (2025-11-12T09:32:35Z)
• Patch Pruning Strategy Based on Robust Statistical Measures of Attention Weight Diversity in Vision Transformers [0.7673339435080445]
  We propose a patch pruning strategy that evaluates the importance of each patch based on the variance of attention weights across multiple attention heads.<n>This approach is inspired by the design of multi-head self-attention, which aims to capture diverse attention patterns across different subspaces of feature representations.
  arXiv  Detail & Related papers  (2025-07-25T11:31:17Z)
• Spatial regularisation for improved accuracy and interpretability in keypoint-based registration [5.286949071316761]
  Recent approaches based on unsupervised keypoint detection stand out as very promising for interpretability.<n>Here, we propose a three-fold loss to regularise the spatial distribution of the features.<n>Our loss considerably improves the interpretability of the features, which now correspond to precise and anatomically meaningful landmarks.
  arXiv  Detail & Related papers  (2025-03-06T14:48:25Z)
• Top-Theta Attention: Sparsifying Transformers by Compensated Thresholding [3.9826635165229223]
  We present Top-Theta (Top-$theta$) Attention, a training-free method for sparsifying transformer attention during inference.<n>Our key insight is that static, per-head thresholds can be calibrated to retain the desired constant number of significant elements per attention row.<n>Top-Theta$ achieves 3-10x reduction in V-cache usage and up to 10x fewer attention elements during inference while no more than 1% in accuracy.
  arXiv  Detail & Related papers  (2025-02-12T12:50:15Z)
• Localized Gaussians as Self-Attention Weights for Point Clouds Correspondence [92.07601770031236]
  We investigate semantically meaningful patterns in the attention heads of an encoder-only Transformer architecture.<n>We find that fixing the attention weights not only accelerates the training process but also enhances the stability of the optimization.
  arXiv  Detail & Related papers  (2024-09-20T07:41:47Z)
• Calibrating Undisciplined Over-Smoothing in Transformer for Weakly Supervised Semantic Segmentation [51.14107156747967]
  Weakly supervised semantic segmentation (WSSS) has attracted considerable attention because it requires fewer annotations than fully supervised approaches.<n>We propose an Adaptive Re-Activation Mechanism (AReAM) to control deep-level attention to undisciplined over-smoothing.<n>AReAM substantially improves segmentation performance compared with existing WSSS methods, reducing noise while sharpening focus on relevant semantic regions.
  arXiv  Detail & Related papers  (2023-05-04T19:11:33Z)
• Patch Selection for Melanoma Classification [1.4631523634811392]
  We investigate two criteria to choose patches: entropy and a spectral similarity criterion. We train a Convolutional Neural Network on the subsets of patches and analyze the training time. We find that, in addition to requiring less preprocessing time, the classifiers trained on the datasets of patches selected based on entropy converge faster than on those selected based on the spectral similarity criterion.
  arXiv  Detail & Related papers  (2022-06-27T20:52:53Z)
• Bayesian Attention Modules [65.52970388117923]
  We propose a scalable version of attention that is easy to implement and optimize. Our experiments show the proposed method brings consistent improvements over the corresponding baselines.
  arXiv  Detail & Related papers  (2020-10-20T20:30:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.