YOSE: You Only Select Essential Tokens for Efficient DiT-based Video Object Removal

• URL: http://arxiv.org/abs/2604.27322v1
• Date: Thu, 30 Apr 2026 02:08:13 GMT
• Title: YOSE: You Only Select Essential Tokens for Efficient DiT-based Video Object Removal
• Abstract summary: YOSE introduces two key components: Batch Variable-length Indexing (BVI) and Diffusion Process Simulator (DiffSim) Module.<n>BVI selects essential tokens based on mask information enabling variable-length token processing across samples.<n>DiffSim provides a diffusion process approximation mechanism for unmasked tokens.<n>Experiments demonstrate YOSE achieves up to 2.5X speedup in 70% of cases.
• Score: 83.78619790270523
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advances in Diffusion Transformer (DiT)-based video generation technologies have shown impressive results for video object removal. However, these methods still suffer from substantial inference latency. For instance, although MiniMax Remover achieves state-of-the-art visual quality, it operates at only around 10FPS, primarily due to dense computations over the entire spatiotemporal token space, even when only a small masked region actually requires processing. In this paper, we present YOSE, You Only Select Essential Tokens, an efficient fine-tuning framework. YOSE introduces two key components: Batch Variable-length Indexing (BVI) and Diffusion Process Simulator (DiffSim) Module. BVI is a differentiable dynamic indexing operator that adaptively selects essential tokens based on mask information, enabling variable-length token processing across samples. DiffSim provides a diffusion process approximation mechanism for unmasked tokens, which simulates the influence of unmasked regions within DiT self-attention to maintain semantic consistency for masked tokens. With these designs, YOSE achieves mask-aware acceleration, where the inference time scales approximately linearly with the masked regions, in contrast to full-token diffusion methods whose computation remains constant regardless of the mask size. Extensive experiments demonstrate that YOSE achieves up to 2.5X speedup in 70% of cases while maintaining visual quality comparable to the baseline. Code is available at: https://github.com/Wucy0519/YOSE-CVPR26.

Related papers

• ST-SimDiff: Balancing Spatiotemporal Similarity and Difference for Efficient Video Understanding with MLLMs [20.712141528369553]
  We develop a training-free framework named ST-Sim-Diff to handle complex video relationships.<n>Our method significantly outperforms state-of-the-art approaches while substantially reducing computational costs.
  arXiv  Detail & Related papers  (2026-05-21T08:27:15Z)
• Edit-Your-Interest: Efficient Video Editing via Feature Most-Similar Propagation [53.05471174430247]
  Edit-Your-Interest is a text-driven, zero-shot video editing method.<n>It reduces computational overhead compared to full-sequence-temporal modeling approaches.<n>It outperforms state-of-the-art methods in both efficiency and visual fidelity.
  arXiv  Detail & Related papers  (2025-10-15T01:55:32Z)
• SimulFlow: Simultaneously Extracting Feature and Identifying Target for Unsupervised Video Object Segmentation [28.19471998380114]
  Unsupervised video object segmentation (UVOS) aims at detecting the primary objects in a given video sequence without any human interposing. Most existing methods rely on two-stream architectures that separately encode the appearance and motion information before fusing them to identify the target and generate object masks. We propose a novel UVOS model called SimulFlow that simultaneously performs feature extraction and target identification.
  arXiv  Detail & Related papers  (2023-11-30T06:44:44Z)
• AiluRus: A Scalable ViT Framework for Dense Prediction [95.1313839257891]
  Vision transformers (ViTs) have emerged as a prevalent architecture for vision tasks owing to their impressive performance. We propose to apply adaptive resolution for different regions in the image according to their importance. We evaluate our proposed method on three different datasets and observe promising performance.
  arXiv  Detail & Related papers  (2023-11-02T12:48:43Z)
• Mask-Free Video Instance Segmentation [102.50936366583106]
  Video masks are tedious and expensive to annotate, limiting the scale and diversity of existing VIS datasets. We propose MaskFreeVIS, achieving highly competitive VIS performance, while only using bounding box annotations for the object state. Our TK-Loss finds one-to-many matches across frames, through an efficient patch-matching step followed by a K-nearest neighbor selection.
  arXiv  Detail & Related papers  (2023-03-28T11:48:07Z)
• AdaMAE: Adaptive Masking for Efficient Spatiotemporal Learning with Masked Autoencoders [44.87786478095987]
  Masked Autoencoders learn general representations for image, text, audio, video, etc., by masked input data from tokens of the visible data. This paper proposes an adaptive masking strategy for MAEs that is end-to-end trainable. AdaMAE samples visible tokens based on the semantic context using an auxiliary sampling network.
  arXiv  Detail & Related papers  (2022-11-16T18:59:48Z)
• SipMask: Spatial Information Preservation for Fast Image and Video Instance Segmentation [149.242230059447]
  We propose a fast single-stage instance segmentation method called SipMask. It preserves instance-specific spatial information by separating mask prediction of an instance to different sub-regions of a detected bounding-box. In terms of real-time capabilities, SipMask outperforms YOLACT with an absolute gain of 3.0% (mask AP) under similar settings.
  arXiv  Detail & Related papers  (2020-07-29T12:21:00Z)
• BlendMask: Top-Down Meets Bottom-Up for Instance Segmentation [103.74690082121079]
  In this work, we achieve improved mask prediction by effectively combining instance-level information with semantic information with lower-level fine-granularity. Our main contribution is a blender module which draws inspiration from both top-down and bottom-up instance segmentation approaches. BlendMask can effectively predict dense per-pixel position-sensitive instance features with very few channels, and learn attention maps for each instance with merely one convolution layer.
  arXiv  Detail & Related papers  (2020-01-02T03:30:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.