Rethinking Depth Pruning for Vision Transformers: A Heterogeneity-Aware Perspective

• URL: http://arxiv.org/abs/2607.03784v1
• Date: Sat, 04 Jul 2026 09:17:25 GMT
• Title: Rethinking Depth Pruning for Vision Transformers: A Heterogeneity-Aware Perspective
• Abstract summary: HetDPT is a heterogeneity-aware depth pruning method that avoids dimension mismatch.<n>HetDPT achieves a 1.58$times$ speedup for DeiT-B while maintaining accuracy and a 1.39$times$ speedup for DeiT-S with nearly no accuracy degradation.<n>When combined with width pruning, HetDPT+ sets a new state-of-the-art record in extreme ViT pruning.
• Score: 12.461911138669848
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While prior studies have successfully compressed vision Transformers (ViTs) through various pruning techniques, most have concentrated on width pruning to achieve significant reductions in model size. Depth pruning, which removes entire layers from a ViT, is notoriously difficult for accuracy recovery despite its potential to deliver higher speedups, limiting the acceleration achieved by existing joint width-and-depth pruning methods. In this work, we reveal that the failure of existing depth pruning methods lies in their neglect of heterogeneity between different layers, and we introduce HetDPT, a heterogeneity-aware depth pruning method that avoids dimension mismatch. Comprehensive experiments on ImageNet-1K, CIFAR-100, COCO, and ADE20K validate our method: HetDPT achieves a 1.58$\times$ speedup for DeiT-B while maintaining accuracy and a 1.39$\times$ speedup for DeiT-S with nearly no accuracy degradation. Furthermore, when combined with width pruning, HetDPT+ sets a new state-of-the-art record in extreme ViT pruning, enhancing the acceleration ratio from 4.24$\times$ to 5.19$\times$ for the Isomorphic-Pruning-2.6G configuration while maintaining near-lossless accuracy; our code is available at https://github.com/Efficient-AI-for-All/HetDPT.

Related papers

• Amped: Adaptive Multi-stage Non-edge Pruning for Edge Detection [53.59823657644554]
  We propose an Adaptive Multi-stage non-edge Pruning framework for Edge Detection(Amped)<n>Amped identifies high-confidence non-edge tokens and removes them as early as possible to substantially reduce computation.<n>To mitigate the structural complexity of existing edge detection networks, we introduce a simple yet high-performance Transformer-based model, termed Streamline Edge Detector(SED)
  arXiv  Detail & Related papers  (2026-03-29T12:26:35Z)
• $D^3$-RSMDE: 40$\times$ Faster and High-Fidelity Remote Sensing Monocular Depth Estimation [72.9912717963138]
  Real-time, high-fidelity monocular depth estimation from remote sensing imagery is crucial for numerous applications.<n>Although using Vision Transformer (ViT) backbones for dense prediction is fast, they often exhibit poor perceptual quality.<n>We propose Depth Detail Diffusion for Remote Sensing Monocular Depth Estimation ($D3$-RSMDE) to overcome these limitations.<n>$D3$-RSMDE achieves a notable 11.85% reduction in the Learned Perceptual Image Patch Similarity (LPIPS) perceptual metric.
  arXiv  Detail & Related papers  (2026-03-17T10:50:36Z)
• FlattenGPT: Depth Compression for Transformer with Layer Flattening [89.7587433008809]
  textbfFlattenGPT is a novel way to detect and reduce depth-wise redundancies.<n>Experiments demonstrate that FlattenGPT enhances model efficiency with a decent trade-off to performance.
  arXiv  Detail & Related papers  (2026-02-09T16:22:58Z)
• Accelerating Vision Transformers with Adaptive Patch Sizes [58.48800204993534]
  Vision Transformers partition input images into uniformly sized patches regardless of their content.<n>We present Adaptive Patch Transformers (APT), which addresses this by using multiple different patch sizes within the same image.<n>APT achieves a drastic speedup in ViT inference and training, increasing throughput by 40% on ViT-L and 50% on ViT-H.
  arXiv  Detail & Related papers  (2025-10-20T20:37:11Z)
• Multi-Dimensional Pruning: Joint Channel, Layer and Block Pruning with Latency Constraint [7.757464614718271]
  Existing pruning approaches are limited to channel pruning and struggle with aggressive parameter reductions. We propose a novel multi-dimensional pruning framework that jointly optimize pruning across channels, layers, and blocks. In 3D object detection, we establish a new state-of-the-art by pruning StreamPETR at a 45% pruning ratio, achieving higher FPS (37.3 vs. 31.7) and mAP (0.451 vs. 0.449) than the dense baseline.
  arXiv  Detail & Related papers  (2024-06-17T20:40:09Z)
• CAIT: Triple-Win Compression towards High Accuracy, Fast Inference, and Favorable Transferability For ViTs [89.79139531731637]
  Vision Transformers (ViTs) have emerged as state-of-the-art models for various vision tasks.<n>We propose a joint underlinecompression method for ViTs that achieves a harmonious blend of high underlineaccuracy, fast underlineinference speed, and favorable underlinetransferability to downstream tasks.
  arXiv  Detail & Related papers  (2023-09-27T16:12:07Z)
• DiffRate : Differentiable Compression Rate for Efficient Vision Transformers [98.33906104846386]
  Token compression aims to speed up large-scale vision transformers (e.g. ViTs) by pruning (dropping) or merging tokens. DiffRate is a novel token compression method that has several appealing properties prior arts do not have.
  arXiv  Detail & Related papers  (2023-05-29T10:15:19Z)
• CAP: Correlation-Aware Pruning for Highly-Accurate Sparse Vision Models [22.055655390093722]
  Correlation Aware Pruner (CAP) significantly pushes the compressibility limits for state-of-the-art architectures. New theoretically-justified pruner handles complex weight correlations accurately and efficiently during the pruning process itself. We show for the first time that extremely-accurate large vision models, trained via self-supervised techniques, can also be pruned to moderate sparsities, with negligible accuracy loss.
  arXiv  Detail & Related papers  (2022-10-14T12:19:09Z)
• Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformer [56.87383229709899]
  We develop an information rectification module (IRM) and a distribution guided distillation scheme for fully quantized vision transformers (Q-ViT) Our method achieves a much better performance than the prior arts.
  arXiv  Detail & Related papers  (2022-10-13T04:00:29Z)
• CP-ViT: Cascade Vision Transformer Pruning via Progressive Sparsity Prediction [16.578899848650675]
  Vision transformer (ViT) has achieved competitive accuracy on a variety of computer vision applications, but its computational cost impedes the deployment on resource-limited mobile devices. We propose a cascade pruning framework named CP-ViT by predicting sparsity in ViT models progressively and dynamically to reduce computational redundancy while minimizing the accuracy loss.
  arXiv  Detail & Related papers  (2022-03-09T08:15:14Z)
• A Unified Pruning Framework for Vision Transformers [40.7622551128182]
  Vision transformer (ViT) and its variants have achieved promising performances in various computer vision tasks. We propose a unified framework for structural pruning of both ViTs and its variants, namely UP-ViTs. Our method focuses on pruning all ViTs components while maintaining the consistency of the model structure.
  arXiv  Detail & Related papers  (2021-11-30T05:01:02Z)
• Global Vision Transformer Pruning with Hessian-Aware Saliency [93.33895899995224]
  This work challenges the common design philosophy of the Vision Transformer (ViT) model with uniform dimension across all the stacked blocks in a model stage. We derive a novel Hessian-based structural pruning criteria comparable across all layers and structures, with latency-aware regularization for direct latency reduction. Performing iterative pruning on the DeiT-Base model leads to a new architecture family called NViT (Novel ViT), with a novel parameter that utilizes parameters more efficiently.
  arXiv  Detail & Related papers  (2021-10-10T18:04:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.