Accelerating Vision Foundation Models with Drop-in Depthwise Convolution

• URL: http://arxiv.org/abs/2605.22132v1
• Date: Thu, 21 May 2026 08:07:23 GMT
• Title: Accelerating Vision Foundation Models with Drop-in Depthwise Convolution
• Abstract summary: We introduce an efficient depthwise convolution-based layer that serves as a drop-in replacement for attention heads.<n>Across both image classification and segmentation tasks, our method achieves 17-20% percent inference speedup with minimal performance degradation.
• Score: 51.50107862675191
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Pretrained vision foundation models deliver strong performance across tasks with limited fine-tuning. However, their Vision Transformer (ViT) backbones impose high inference costs, limiting deployment on resource-constrained devices. In this work, we accelerate large-scale pretrained ViTs while preserving their feature extraction capabilities by exploiting the intrinsic convolution-like behavior of some attention heads. Specifically, we introduce an efficient depthwise convolution-based layer that serves as a drop-in replacement for these heads. Additionally, we propose simple strategies to identify which heads can be replaced and introduce a fine-tuning procedure that recovers downstream task performance. Across both image classification and segmentation tasks, our method achieves 17-20\% percent inference speedup with minimal performance degradation. We validate the approach through detailed derivations, extensive experiments, and efficiency benchmarks. The reference implementation is publicly available.

Related papers

• Rethinking Layer-Wise Information Allocation for Vision Foundation Model Adaptation [42.92624141712034]
  Prompted Information Bottlenecks (PIB) is a framework that regularizes layer-wise compression-sufficiency trade-offs and promotes a more coherent cross-layer information path.<n>PIB achieves strong performance across 34 datasets, reaching 92.1% on FGVC, 93.01% on HTA, and 77.33% on VTAB-1k.<n>These results position PIB as both a practical method and an information-allocation perspective for adapting frozen vision foundation models.
  arXiv  Detail & Related papers  (2026-07-24T04:39:55Z)
• ViT-FREE: Efficient Face Recognition via Early Exiting and Synthetic Adaptation [16.250163286204593]
  Vision Transformers (ViTs) have gained significant attention in computer vision and shown strong potential for face recognition (FR)<n>In this work, we investigate early exiting in pretrained ViTs as a simple yet effective training-free strategy for efficient FR inference.<n>We introduce ViT-FREE, a multi-exit framework that enables face verification directly from intermediate representations without modifying or retraining the backbone model.
  arXiv  Detail & Related papers  (2026-06-10T12:46:39Z)
• FREE: Fast and Robust Vision Language Models with Early Exits [5.402030962296633]
  We introduce FREE, an adversarial training approach within a GAN-based framework.<n>Our method focuses on performing input-adaptive inference that increases inference speed with minimal drop in performance.<n>We experimentally validate that our method speeds up the inference process by more than 1.51x while retaining comparable performance.
  arXiv  Detail & Related papers  (2025-06-07T18:26:58Z)
• Dyn-Adapter: Towards Disentangled Representation for Efficient Visual Recognition [22.615830919860777]
  This paper presents an efficient visual recognition paradigm, called Dynamic Adapter (Dyn-Adapter) We devise a dynamic architecture with balanced early heads for multi-level feature extraction, along with adaptive training strategy. We reduce FLOPs during inference by 50%, while maintaining or even yielding higher recognition accuracy.
  arXiv  Detail & Related papers  (2024-07-19T13:33:38Z)
• Parameter-Efficient and Memory-Efficient Tuning for Vision Transformer: A Disentangled Approach [87.8330887605381]
  We show how to adapt a pre-trained Vision Transformer to downstream recognition tasks with only a few learnable parameters. We synthesize a task-specific query with a learnable and lightweight module, which is independent of the pre-trained model. Our method achieves state-of-the-art performance under memory constraints, showcasing its applicability in real-world situations.
  arXiv  Detail & Related papers  (2024-07-09T15:45:04Z)
• Visual Prompt Tuning in Null Space for Continual Learning [51.96411454304625]
  Existing prompt-tuning methods have demonstrated impressive performances in continual learning (CL) This paper aims to learn each task by tuning the prompts in the direction orthogonal to the subspace spanned by previous tasks' features. In practice, an effective null-space-based approximation solution has been proposed to implement the prompt gradient projection.
  arXiv  Detail & Related papers  (2024-06-09T05:57:40Z)
• MoE-FFD: Mixture of Experts for Generalized and Parameter-Efficient Face Forgery Detection [54.545054873239295]
  Deepfakes have recently raised significant trust issues and security concerns among the public.<n>ViT-based methods take advantage of the expressivity of transformers, achieving superior detection performance.<n>This work introduces Mixture-of-Experts modules for Face Forgery Detection (MoE-FFD), a generalized yet parameter-efficient ViT-based approach.
  arXiv  Detail & Related papers  (2024-04-12T13:02:08Z)
• Hierarchical Side-Tuning for Vision Transformers [33.536948382414316]
  Fine-tuning pre-trained Vision Transformers (ViTs) has showcased significant promise in enhancing visual recognition tasks. PETL has shown potential for achieving high performance with fewer parameter updates compared to full fine-tuning. This paper introduces Hierarchical Side-Tuning (HST), an innovative PETL method facilitating the transfer of ViT models to diverse downstream tasks.
  arXiv  Detail & Related papers  (2023-10-09T04:16:35Z)
• BUS:Efficient and Effective Vision-language Pre-training with Bottom-Up Patch Summarization [89.52943129132217]
  We propose a Bottom-Up Patch Summarization approach named BUS to learn a concise summary of lengthy visual token sequences efficiently. We incorporate a Text-Semantics-Aware Patch Selector (TSPS) into the ViT backbone to perform a coarse-grained visual token extraction. This bottom-up collaboration enables our BUS to yield high training efficiency while maintaining or even improving effectiveness.
  arXiv  Detail & Related papers  (2023-07-17T14:08:17Z)
• Pro-tuning: Unified Prompt Tuning for Vision Tasks [133.12978197265596]
  Fine-tuning is the de-facto approach to leverage pre-trained vision models to perform downstream tasks. In this work, we propose parameter-efficient Prompt tuning (Pro-tuning) to adapt frozen vision models to various downstream vision tasks.
  arXiv  Detail & Related papers  (2022-07-28T21:09:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.