When Token Compression Breaks: Structural Pruning vs. Token Reduction for Robust ViT Segmentation under High Compression

• URL: http://arxiv.org/abs/2607.02237v1
• Date: Thu, 02 Jul 2026 14:34:31 GMT
• Title: When Token Compression Breaks: Structural Pruning vs. Token Reduction for Robust ViT Segmentation under High Compression
• Abstract summary: Vision Transformers (ViTs) are strong backbones for semantic segmentation, but their computational cost limits deployment.<n>Recent token compression methods for efficient ViT-based segmentation reduce this cost by decreasing the number of tokens.<n>We benchmark token compression and structural pruning methods for ViT-based semantic segmentation under matched FLOPs.
• Score: 22.74689289299356
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision Transformers (ViTs) are strong backbones for semantic segmentation, but their computational cost limits deployment. Recent token compression methods for efficient transformer-based segmentation reduce this cost by decreasing the number of tokens. However, existing evaluations primarily focus on low-to-moderate compression, leaving their behavior under aggressive compression and corrupted inputs unclear. Meanwhile, structural pruning provides an orthogonal route to efficiency by removing redundant components in the ViT architecture, but is rarely compared to token compression under a unified protocol. To bridge this gap, we benchmark representative token compression and structural pruning methods for ViT-based semantic segmentation under matched FLOPs on ADE20K and Cityscapes, together with their common-corruption variants ADE20K-C and Cityscapes-C. Our results reveal a consistent trend on both clean and corrupted inputs: token compression is highly effective at mild reductions but degrades sharply when compression becomes severe, consistent with substantial information loss from overly aggressive token reduction. In contrast, structural pruning exhibits a smoother degradation curve and is more stable at high compression. Motivated by these findings, we study a prune-then-merge pipeline that applies moderate token compression on top of a moderately pruned backbone. At comparable FLOPs, this combined strategy consistently achieves a better accuracy-robustness trade-off at high compression, offering a practical recipe for deployment-oriented ViT segmentation. Code is available at https://github.com/phatnguyencs/vit-seg-compression.

Related papers

• Beyond Single-Dimensional Compression: The Compound Sparsity Frontier of Large Language Models [6.94102129809152]
  Large language models (LLMs) are often compressed through static parameter pruning or dynamic token-level computation.<n>We study a minimalist compound sparsity framework that first applies low-rank approximation and channel pruning to obtain a statically compressed backbone.
  arXiv  Detail & Related papers  (2026-06-29T01:33:30Z)
• Compression as an Adversarial Amplifier Through Decision Space Reduction [14.526561120852532]
  We study a previously unexplored adversarial setting in which attacks are applied directly in compressed representations.<n>Under identical nominal perturbation budgets, compression-aware attacks are substantially more effective than their pixel-space counterparts.<n>We attribute this effect to decision space reduction, whereby compression induces a non-invertible, information-losing transformation.
  arXiv  Detail & Related papers  (2026-04-08T11:15:45Z)
• EchoKV: Efficient KV Cache Compression via Similarity-Based Reconstruction [55.026048429595384]
  EchoKV is a flexible KV cache compression scheme that enables on-demand transitions between standard and compressed inference.<n>We introduce a two-stage fine-tuning strategy that allows for rapid, low-cost training.
  arXiv  Detail & Related papers  (2026-03-24T07:58:42Z)
• Arbitrary Ratio Feature Compression via Next Token Prediction [52.10426317889982]
  Arbitrary Ratio Feature Compression (ARFC) framework supports any compression ratio with a single model.<n>ARC is an auto-regressive model that performs compression via next-gressive prediction.<n>MoS module refines the compressed tokens by utilizing multiple compression results.<n>ERGC is integrated into the training process to preserve semantic and structural relationships during compression.
  arXiv  Detail & Related papers  (2026-02-12T02:38:57Z)
• On the Adversarial Robustness of Large Vision-Language Models under Visual Token Compression [22.436953683970007]
  We show that existing encoder-based attacks can substantially overestimate the robustness of compressed vision-language models (LVLMs)<n>We propose the Compression-AliGnEd attack (CAGE), which aligns perturbation optimization with compression inference without assuming access to the deployed compression mechanism or its token budget.
  arXiv  Detail & Related papers  (2026-01-29T10:47:21Z)
• Efficient Token Compression for Vision Transformer with Spatial Information Preserved [59.79302182800274]
  Token compression is essential for reducing the computational and memory requirements of transformer models.<n>We propose an efficient and hardware-compatible token compression method called Prune and Merge.
  arXiv  Detail & Related papers  (2025-03-30T14:23:18Z)
• UniPCGC: Towards Practical Point Cloud Geometry Compression via an Efficient Unified Approach [4.754973569457509]
  We propose an efficient unified point cloud geometry compression framework, dubbed as UniPCGC.<n>It supports lossy compression, lossless compression, variable rate and variable complexity.<n>Our method achieves a compression ratio (CR) gain of 8.1% on lossless compression, and a Bjontegaard Delta Rate (BD-Rate) gain of 14.02% on lossy compression.
  arXiv  Detail & Related papers  (2025-03-24T10:51:28Z)
• Token Compensator: Altering Inference Cost of Vision Transformer without Re-Tuning [63.43972993473501]
  Token compression expedites the training and inference of Vision Transformers (ViTs) However, when applied to downstream tasks, compression degrees are mismatched between training and inference stages. We propose a model arithmetic framework to decouple the compression degrees between the two stages.
  arXiv  Detail & Related papers  (2024-08-13T10:36:43Z)
• DiffRate : Differentiable Compression Rate for Efficient Vision Transformers [98.33906104846386]
  Token compression aims to speed up large-scale vision transformers (e.g. ViTs) by pruning (dropping) or merging tokens. DiffRate is a novel token compression method that has several appealing properties prior arts do not have.
  arXiv  Detail & Related papers  (2023-05-29T10:15:19Z)
• Towards Compact CNNs via Collaborative Compression [166.86915086497433]
  We propose a Collaborative Compression scheme, which joints channel pruning and tensor decomposition to compress CNN models. We achieve 52.9% FLOPs reduction by removing 48.4% parameters on ResNet-50 with only a Top-1 accuracy drop of 0.56% on ImageNet 2012.
  arXiv  Detail & Related papers  (2021-05-24T12:07:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.