FIS-DiT: Breaking the Few-Step Video Inference Barrier via Training-Free Frame Interleaved Sparsity

• URL: http://arxiv.org/abs/2605.11869v1
• Date: Tue, 12 May 2026 09:49:36 GMT
• Title: FIS-DiT: Breaking the Few-Step Video Inference Barrier via Training-Free Frame Interleaved Sparsity
• Abstract summary: Video Diffusion Transformers (DiTs) can be substantially reduced through model, but per-step inference latency remains a critical bottleneck.<n>We propose Frame Inter Sparsity DiT (FIS-DiT), a training-free and operator-agnostic framework that shifts the optimization focus from the temporal trajectory to the latent frame position.<n>FIS-DiT consistently achieves 2.11--2.41$times$ speedup with negligible degradation across VBench-Q and CLIP metrics.
• Score: 23.184639887235218
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While the overall inference latency of Video Diffusion Transformers (DiTs) can be substantially reduced through model distillation, per-step inference latency remains a critical bottleneck. Existing acceleration paradigms primarily exploit redundancy across the denoising trajectory; however, we identify a limitation where these step-wise strategies encounter diminishing returns in few-step regimes. In such scenarios, the scarcity of temporal states prevents effective feature reuse or predictive modeling, creating a formidable barrier to further acceleration. To overcome this, we propose Frame Interleaved Sparsity DiT (FIS-DiT), a training-free and operator-agnostic framework that shifts the optimization focus from the temporal trajectory to the latent frame dimension. Our approach is motivated by an intrinsic duality within this dimension: the existence of frame-wise sparsity that permits reduced computation, coupled with a structural consistency where each frame position remains equally vital to the global spatiotemporal context. Leveraging this insight, we implement Frame Interleaved Sparsity (FIS) as an execution strategy that manipulates frame subsets across the model hierarchy, refreshing all latent positions without requiring full-scale block computation. Empirical evaluations on Wan 2.2 and HunyuanVideo 1.5 demonstrate that FIS-DiT consistently achieves 2.11--2.41$\times$ speedup with negligible degradation across VBench-Q and CLIP metrics, providing a scalable and robust pathway toward real-time high-definition video generation.

Related papers

• COMET: Contrastive Motion-Enhanced Temporal Reasoning for Video Multimodal Large Language Models [15.39037309359101]
  Video multimodal large language models have advanced significantly, yet fine-grained motion-temporal understanding remains fragile.<n>We propose COMET, a temporally grounded framework that systematically strengthens video MLLMs through explicit temporal representation.<n>For optimization, COMET combines temporal prior distillation with a forward-reverse TC-GRPO stage that turns temporal order into a direct learning signal.
  arXiv  Detail & Related papers  (2026-08-21T12:28:36Z)
• ST-Prune: Training-Free Spatio-Temporal Token Pruning for Vision-Language Models in Autonomous Driving [31.688411695647357]
  We propose ST-Prune, a training-free, plug-and-play framework comprising two complementary modules: Motionaware RSP (MTP) and Ring-view Spatial Pruning (RSPRSP)<n>These two modules constitute a complete-temporal pruning process, preserving key geometry-the-art for training-free token pruning.<n> ST-Prune achieves near-lossless performance with certain metrics surpassing the full-model baseline, while maintaining speeds comparable to existing pruning approaches.
  arXiv  Detail & Related papers  (2026-04-21T06:51:08Z)
• D$^2$-VR: Degradation-Robust and Distilled Video Restoration with Synergistic Optimization Strategy [7.553742541566094]
  integration of diffusion priors with temporal alignment has emerged as a transformative paradigm for video restoration, delivering fantastic perceptual quality.<n>We propose textbfD$2$-VR, a single-image diffusion-based video-restoration framework with low-step inference.
  arXiv  Detail & Related papers  (2026-02-09T08:52:51Z)
• Efficient On-Board Processing of Oblique UAV Video for Rapid Flood Extent Mapping [7.460695517551536]
  Temporal Token Reuse (TTR) is an adaptive inference framework capable of accelerating video segmentation on embedded devices.<n>We show that TTR achieves a 30% reduction in inference latency with negligible degradation in segmentation accuracy ( 0.5% mIoU)<n>These findings confirm that TTR effectively shifts the operational frontier, enabling high-fidelity, real-time oblique video understanding for time-critical remote sensing missions.
  arXiv  Detail & Related papers  (2026-01-16T13:41:56Z)
• Error-Propagation-Free Learned Video Compression With Dual-Domain Progressive Temporal Alignment [92.57576987521107]
  We propose a novel unifiedtransform framework with dual-domain progressive temporal alignment and quality-conditioned mixture-of-expert (QCMoE)<n>QCMoE allows continuous and consistent rate control with appealing R-D performance.<n> Experimental results show that the proposed method achieves competitive R-D performance compared with the state-of-the-arts.
  arXiv  Detail & Related papers  (2025-12-11T09:14:51Z)
• Bidirectional Feature-aligned Motion Transformation for Efficient Dynamic Point Cloud Compression [97.66080040613726]
  We propose a Bidirectional Feature-aligned Motion Transformation (Bi-FMT) framework that implicitly models motion in the feature space.<n>Bi-FMT aligns features across both past and future frames to produce temporally consistent latent representations.<n>We show Bi-FMT surpasses D-DPCC and AdaDPCC in both compression efficiency and runtime.
  arXiv  Detail & Related papers  (2025-09-18T03:51:06Z)
• FCA2: Frame Compression-Aware Autoencoder for Modular and Fast Compressed Video Super-Resolution [68.77813885751308]
  State-of-the-art (SOTA) compressed video super-resolution (CVSR) models face persistent challenges, including prolonged inference time, complex training pipelines, and reliance on auxiliary information.<n>We propose an efficient and scalable solution inspired by the structural and statistical similarities between hyperspectral images (HSI) and video data.<n>Our approach introduces a compression-driven dimensionality reduction strategy that reduces computational complexity, accelerates inference, and enhances the extraction of temporal information across frames.
  arXiv  Detail & Related papers  (2025-06-13T07:59:52Z)
• Enhancing Low-Cost Video Editing with Lightweight Adaptors and Temporal-Aware Inversion [28.38307253613529]
  We propose a framework that integrates temporal-spatial and semantic consistency with Baliteral DDIM inversion.<n>Our method significantly improves perceptual quality, text-image alignment, and temporal coherence, as demonstrated on the MSR-VTT dataset.
  arXiv  Detail & Related papers  (2025-01-08T16:41:31Z)
• Low-Light Video Enhancement via Spatial-Temporal Consistent Decomposition [52.89441679581216]
  Low-Light Video Enhancement (LLVE) seeks to restore dynamic or static scenes plagued by severe invisibility and noise.<n>We present an innovative video decomposition strategy that incorporates view-independent and view-dependent components.<n>Our framework consistently outperforms existing methods, establishing a new SOTA performance.
  arXiv  Detail & Related papers  (2024-05-24T15:56:40Z)
• Cross-Consistent Deep Unfolding Network for Adaptive All-In-One Video Restoration [78.14941737723501]
  We propose a Cross-consistent Deep Unfolding Network (CDUN) for All-In-One VR. By orchestrating two cascading procedures, CDUN achieves adaptive processing for diverse degradations. In addition, we introduce a window-based inter-frame fusion strategy to utilize information from more adjacent frames.
  arXiv  Detail & Related papers  (2023-09-04T14:18:00Z)
• TimeLens: Event-based Video Frame Interpolation [54.28139783383213]
  We introduce Time Lens, a novel indicates equal contribution method that leverages the advantages of both synthesis-based and flow-based approaches. We show an up to 5.21 dB improvement in terms of PSNR over state-of-the-art frame-based and event-based methods.
  arXiv  Detail & Related papers  (2021-06-14T10:33:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.