Video Patch Pruning: Efficient Video Instance Segmentation via Early Token Reduction

• URL: http://arxiv.org/abs/2604.00827v1
• Date: Wed, 01 Apr 2026 12:36:32 GMT
• Title: Video Patch Pruning: Efficient Video Instance Segmentation via Early Token Reduction
• Abstract summary: We present a novel Video Patch Pruning framework (VPP) that integrates temporal prior knowledge to enable efficient sparsity within early ViT layers.<n>VPP excels the high-sparsity regime, sustaining remarkable performance even when patch usage is reduced below 55%.<n> Specifically, it preserves stable results with a maximal performance drop of 0.6% on the Youtube-VIS 2021 dataset.
• Score: 24.23084511204264
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision Transformers (ViTs) have demonstrated state-ofthe-art performance in several benchmarks, yet their high computational costs hinders their practical deployment. Patch Pruning offers significant savings, but existing approaches restrict token reduction to deeper layers, leaving early-stage compression unexplored. This limits their potential for holistic efficiency. In this work, we present a novel Video Patch Pruning framework (VPP) that integrates temporal prior knowledge to enable efficient sparsity within early ViT layers. Our approach is motivated by the observation that prior features extracted from deeper layers exhibit strong foreground selectivity. Therefore we propose a fully differentiable module for temporal mapping to accurately select the most relevant patches in early network stages. Notably, the proposed method enables a patch reduction of up to 60% in dense prediction tasks, exceeding the capabilities of conventional image-based patch pruning, which typically operate around a 30% patch sparsity. VPP excels the high-sparsity regime, sustaining remarkable performance even when patch usage is reduced below 55%. Specifically, it preserves stable results with a maximal performance drop of 0.6% on the Youtube-VIS 2021 dataset.

Related papers

• ViT-FREE: Efficient Face Recognition via Early Exiting and Synthetic Adaptation [16.250163286204593]
  Vision Transformers (ViTs) have gained significant attention in computer vision and shown strong potential for face recognition (FR)<n>In this work, we investigate early exiting in pretrained ViTs as a simple yet effective training-free strategy for efficient FR inference.<n>We introduce ViT-FREE, a multi-exit framework that enables face verification directly from intermediate representations without modifying or retraining the backbone model.
  arXiv  Detail & Related papers  (2026-06-10T12:46:39Z)
• Accelerating Vision Foundation Models with Drop-in Depthwise Convolution [51.50107862675191]
  We introduce an efficient depthwise convolution-based layer that serves as a drop-in replacement for attention heads.<n>Across both image classification and segmentation tasks, our method achieves 17-20% percent inference speedup with minimal performance degradation.
  arXiv  Detail & Related papers  (2026-05-21T08:07:23Z)
• Temporal Aware Pruning for Efficient Diffusion-based Video Generation [22.469564305396002]
  We propose TAPE, a training-free Temporal Aware for Efficient diffusion-based video generation.<n>We show that TAPE delivers significant speedups while preserving high visual fidelity, outperforming prior token reduction approaches.
  arXiv  Detail & Related papers  (2026-05-18T04:18:24Z)
• Training-free Latent Inter-Frame Pruning with Attention Recovery [50.889009147480856]
  Current video generation models suffer from high computational latency, making real-time applications prohibitively costly.<n>We propose the Latent Inter-frame Pruning with Attention Recovery (LIPAR) framework, which detects and skips recomputing duplicated latent patches.<n>Our method increases video editing throughput by $1.45times$, on average achieving 12.2 FPS on an NVIDIA A6000.
  arXiv  Detail & Related papers  (2026-03-06T01:49:47Z)
• A Glimpse to Compress: Dynamic Visual Token Pruning for Large Vision-Language Models [94.49953824684853]
  We introduce a dynamic pruning framework, GlimpsePrune, inspired by human cognition.<n>It takes a data-driven ''glimpse'' and prunes irrelevant visual tokens in a single forward pass before answer generation.<n>An enhanced GlimpsePrune+ achieves 110% of the baseline performance while maintaining a similarly high pruning rate.
  arXiv  Detail & Related papers  (2025-08-03T02:15:43Z)
• PrefixKV: Adaptive Prefix KV Cache is What Vision Instruction-Following Models Need for Efficient Generation [97.41972925670508]
  Large vision-language models (LVLMs) incur significant computational and memory overhead during inference.<n>We present PrefixKV, where "Prefix" means the top-ranked KV based on importance rather than position in the original sequence.<n>Our method achieves the state-of-the-art performance compared with others.
  arXiv  Detail & Related papers  (2024-12-04T15:48:59Z)
• Faster Image2Video Generation: A Closer Look at CLIP Image Embedding's Impact on Spatio-Temporal Cross-Attentions [27.111140222002653]
  This paper investigates the role of CLIP image embeddings within the Stable Video Diffusion (SVD) framework. We introduce the VCUT, a training-free approach optimized for efficiency within the SVD architecture. The implementation of VCUT leads to a reduction of up to 322T Multiple-Accumulate Operations (MACs) per video and a decrease in model parameters by up to 50M, achieving a 20% reduction in latency compared to the baseline.
  arXiv  Detail & Related papers  (2024-07-27T08:21:14Z)
• PaPr: Training-Free One-Step Patch Pruning with Lightweight ConvNets for Faster Inference [11.112356346406365]
  PaPr is a method for substantially pruning redundant patches with minimal accuracy loss using lightweight ConvNets. It achieves significantly higher accuracy over state-of-the-art patch reduction methods with similar FLOP count reduction.
  arXiv  Detail & Related papers  (2024-03-24T05:50:00Z)
• EcoTTA: Memory-Efficient Continual Test-time Adaptation via Self-distilled Regularization [71.70414291057332]
  TTA may primarily be conducted on edge devices with limited memory. Long-term adaptation often leads to catastrophic forgetting and error accumulation. We present lightweight meta networks that can adapt the frozen original networks to the target domain.
  arXiv  Detail & Related papers  (2023-03-03T13:05:30Z)
• Patch Slimming for Efficient Vision Transformers [107.21146699082819]
  We study the efficiency problem for visual transformers by excavating redundant calculation in given networks. We present a novel patch slimming approach that discards useless patches in a top-down paradigm. Experimental results on benchmark datasets demonstrate that the proposed method can significantly reduce the computational costs of vision transformers.
  arXiv  Detail & Related papers  (2021-06-05T09:46:00Z)
• Adaptive Focus for Efficient Video Recognition [29.615394426035074]
  We propose a reinforcement learning based approach for efficient spatially adaptive video recognition (AdaFocus) A light-weighted ConvNet is first adopted to quickly process the full video sequence, whose features are used by a recurrent policy network to localize the most task-relevant regions. During offline inference, once the informative patch sequence has been generated, the bulk of computation can be done in parallel, and is efficient on modern GPU devices.
  arXiv  Detail & Related papers  (2021-05-07T13:24:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.