HAViT: Historical Attention Vision Transformer

• URL: http://arxiv.org/abs/2603.18585v1
• Date: Thu, 19 Mar 2026 07:46:47 GMT
• Title: HAViT: Historical Attention Vision Transformer
• Abstract summary: Vision Transformers have excelled in computer vision but their attention mechanisms operate independently across layers.<n>We propose a cross-layer attention propagation method that preserves and integrates historical attention matrices.<n>This approach enables progressive refinement of attention patterns throughout the transformer hierarchy.
• Score: 7.419725234099727
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision Transformers have excelled in computer vision but their attention mechanisms operate independently across layers, limiting information flow and feature learning. We propose an effective cross-layer attention propagation method that preserves and integrates historical attention matrices across encoder layers, offering a principled refinement of inter-layer information flow in Vision Transformers. This approach enables progressive refinement of attention patterns throughout the transformer hierarchy, enhancing feature acquisition and optimization dynamics. The method requires minimal architectural changes, adding only attention matrix storage and blending operations. Comprehensive experiments on CIFAR-100 and TinyImageNet demonstrate consistent accuracy improvements, with ViT performance increasing from 75.74% to 77.07% on CIFAR-100 (+1.33%) and from 57.82% to 59.07% on TinyImageNet (+1.25%). Cross-architecture validation shows similar gains across transformer variants, with CaiT showing 1.01% enhancement. Systematic analysis identifies the blending hyperparameter of historical attention (alpha = 0.45) as optimal across all configurations, providing the ideal balance between current and historical attention information. Random initialization consistently outperforms zero initialization, indicating that diverse initial attention patterns accelerate convergence and improve final performance. Our code is publicly available at https://github.com/banik-s/HAViT.

Related papers

• A Transformer-in-Transformer Network Utilizing Knowledge Distillation for Image Recognition [0.8196125054032961]
  We propose an inner-outer transformer-based architecture, which gives attention to the global and local aspects of the image.<n>Our approach enhances learning efficiency and effectiveness.<n>Remarkably, the proposed Transformer-in-Transformer Network (TITN) model achieves impressive milestones across various datasets.
  arXiv  Detail & Related papers  (2025-02-24T00:41:46Z)
• SpectFormer: Frequency and Attention is what you need in a Vision Transformer [28.01996628113975]
  Vision transformers have been applied successfully for image recognition tasks. We hypothesize that both spectral and multi-headed attention plays a major role. We propose the novel Spectformer architecture for transformers that combines spectral and multi-headed attention layers.
  arXiv  Detail & Related papers  (2023-04-13T12:27:17Z)
• Skip-Attention: Improving Vision Transformers by Paying Less Attention [55.47058516775423]
  Vision computation transformers (ViTs) use expensive self-attention operations in every layer. We propose SkipAt, a method to reuse self-attention from preceding layers to approximate attention at one or more subsequent layers. We show the effectiveness of our method in image classification and self-supervised learning on ImageNet-1K, semantic segmentation on ADE20K, image denoising on SIDD, and video denoising on DAVIS.
  arXiv  Detail & Related papers  (2023-01-05T18:59:52Z)
• A Close Look at Spatial Modeling: From Attention to Convolution [70.5571582194057]
  Vision Transformers have shown great promise recently for many vision tasks due to the insightful architecture design and attention mechanism. We generalize self-attention formulation to abstract a queryirrelevant global context directly and integrate the global context into convolutions. With less than 14M parameters, our FCViT-S12 outperforms related work ResT-Lite by 3.7% top1 accuracy on ImageNet-1K.
  arXiv  Detail & Related papers  (2022-12-23T19:13:43Z)
• Vicinity Vision Transformer [53.43198716947792]
  We present a Vicinity Attention that introduces a locality bias to vision transformers with linear complexity. Our approach achieves state-of-the-art image classification accuracy with 50% fewer parameters than previous methods.
  arXiv  Detail & Related papers  (2022-06-21T17:33:53Z)
• Transformer Tracking with Cyclic Shifting Window Attention [17.73494432795304]
  We propose a new transformer architecture with multi-scale cyclic shifting window attention for visual object tracking. In this paper, we demonstrate the superior performance of our method, which also sets the new state-of-the-art records on five challenging datasets.
  arXiv  Detail & Related papers  (2022-05-08T07:46:34Z)
• Attribute Surrogates Learning and Spectral Tokens Pooling in Transformers for Few-shot Learning [50.95116994162883]
  Vision transformers have been thought of as a promising alternative to convolutional neural networks for visual recognition. This paper presents hierarchically cascaded transformers that exploit intrinsic image structures through spectral tokens pooling. HCTransformers surpass the DINO baseline by a large margin of 9.7% 5-way 1-shot accuracy and 9.17% 5-way 5-shot accuracy on miniImageNet.
  arXiv  Detail & Related papers  (2022-03-17T03:49:58Z)
• BViT: Broad Attention based Vision Transformer [13.994231768182907]
  We propose the broad attention to improve the performance by incorporating the attention relationship of different layers for vision transformer, which is called BViT. Experiments on image classification tasks demonstrate that BViT delivers state-of-the-art accuracy of 74.8%/81.6% top-1 accuracy on ImageNet with 5M/22M parameters.
  arXiv  Detail & Related papers  (2022-02-13T09:23:29Z)
• CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows [99.36226415086243]
  We present CSWin Transformer, an efficient and effective Transformer-based backbone for general-purpose vision tasks. A challenging issue in Transformer design is that global self-attention is very expensive to compute whereas local self-attention often limits the field of interactions of each token.
  arXiv  Detail & Related papers  (2021-07-01T17:59:56Z)
• XCiT: Cross-Covariance Image Transformers [73.33400159139708]
  We propose a "transposed" version of self-attention that operates across feature channels rather than tokens. The resulting cross-covariance attention (XCA) has linear complexity in the number of tokens, and allows efficient processing of high-resolution images.
  arXiv  Detail & Related papers  (2021-06-17T17:33:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.