Training-Free Sparse Attention for Fast Video Generation via Offline Layer-Wise Sparsity Profiling and Online Bidirectional Co-Clustering

• URL: http://arxiv.org/abs/2603.18636v1
• Date: Thu, 19 Mar 2026 09:00:08 GMT
• Title: Training-Free Sparse Attention for Fast Video Generation via Offline Layer-Wise Sparsity Profiling and Online Bidirectional Co-Clustering
• Abstract summary: Diffusion Transformers (DiTs) achieve strong video generation quality but suffer from high inference cost due to dense 3D attention.<n>We propose SVOO, a training-free Sparse attention framework for fast Video generation via Offline layer-wise sparsity profiling and Online bidirectional co-clustering.
• Score: 32.40731465958384
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Diffusion Transformers (DiTs) achieve strong video generation quality but suffer from high inference cost due to dense 3D attention, leading to the development of sparse attention technologies to improve efficiency. However, existing training-free sparse attention methods in video generation still face two unresolved limitations: ignoring layer heterogeneity in attention pruning and ignoring query-key coupling in block partitioning, which hinder a better quality-speedup trade-off. In this work, we uncover a critical insight that the attention sparsity of each layer is its intrinsic property, with minor effects across different inputs. Motivated by this, we propose SVOO, a training-free Sparse attention framework for fast Video generation via Offline layer-wise sparsity profiling and Online bidirectional co-clustering. Specifically, SVOO adopts a two-stage paradigm: (i) offline layer-wise sensitivity profiling to derive intrinsic per-layer pruning levels, and (ii) online block-wise sparse attention via a novel bidirectional co-clustering algorithm. Extensive experiments on seven widely used video generation models demonstrate that SVOO achieves a superior quality-speedup trade-off over state-of-the-art methods, delivering up to $1.93\times$ speedup while maintaining a PSNR of up to 29 dB on Wan2.1.

Related papers

• PARE: Pruning and Adaptive Routing for Efficient Video Generation [71.54959622788608]
  Video Diffusion Transformers (DiTs) generate high-quality videos but demand substantial compute due to wide blocks, deep architectures, and iterative sampling.<n>Recent methods reduce cost by compressing width, depth, or sampling steps, but typically commit to a fixed architecture that cannot adapt to individual inputs or denoising stages.<n>We propose PARE, which jointly compresses width and depth with structure-aware pruning and input-adaptive routing.
  arXiv  Detail & Related papers  (2026-05-26T17:43:25Z)
• HASTE: Training-Free Video Diffusion Acceleration via Head-Wise Adaptive Sparse Attention [68.95533683996236]
  Training-free sparse attention is attractive because it accelerates pretrained models without retraining.<n>Existing online top-$p$ sparse attention still spends non-negligible cost on mask prediction.<n>We show that these two overlooked factors limit the practical speed-quality trade-off of training-free sparse attention in Video DiTs.
  arXiv  Detail & Related papers  (2026-05-14T07:57:55Z)
• Free-Lunch Long Video Generation via Layer-Adaptive O.O.D Correction [12.736570247861309]
  We propose FreeLOC, a training-free, layer-adaptive framework for long-video inference.<n>Our approach significantly outperforms existing training-free methods, achieving state-of-the-art results in both temporal consistency and visual quality.
  arXiv  Detail & Related papers  (2026-03-26T09:12:14Z)
• SALAD: Achieve High-Sparsity Attention via Efficient Linear Attention Tuning for Video Diffusion Transformer [58.79642223409644]
  Diffusion Transformers have recently demonstrated remarkable performance in video generation.<n>We propose SALAD, introducing a lightweight linear attention branch in parallel with the sparse attention.<n>Our method attains 90% sparsity and 1.72x inference speedup, while maintaining generation quality comparable to the full attention baseline.
  arXiv  Detail & Related papers  (2026-01-23T07:28:53Z)
• USV: Unified Sparsification for Accelerating Video Diffusion Models [11.011602744993942]
  Unified Sparsification for Video diffusion models is an end-to-end trainable framework.<n>It orchestrates sparsification across both the model's internal computation and its sampling process.<n>It achieves up to 83.3% speedup in the denoising process and 22.7% end-to-end acceleration, while maintaining high visual fidelity.
  arXiv  Detail & Related papers  (2025-12-05T14:40:06Z)
• LinVideo: A Post-Training Framework towards O(n) Attention in Efficient Video Generation [19.69237169218592]
  We present an efficient data-free post-training framework that replaces target number of self-attention modules with linear attention.<n>Our method achieves a 1.25-2.00x speedup while preserving generation quality and our 4-step distilled model further delivers a 15.92x latency reduction with minimal visual quality drop.
  arXiv  Detail & Related papers  (2025-10-09T15:03:39Z)
• Attention Surgery: An Efficient Recipe to Linearize Your Video Diffusion Transformer [13.545000689565732]
  Transformer-based video diffusion models (VDMs) deliver state-of-the-art video generation quality but are constrained by the quadratic cost of self-attention.<n>We introduce Attention Surgery, an efficient framework for linearizing or hybridizing attention in pretrained VDMs without training from scratch.
  arXiv  Detail & Related papers  (2025-09-29T15:09:51Z)
• Lightning Fast Caching-based Parallel Denoising Prediction for Accelerating Talking Head Generation [50.04968365065964]
  Diffusion-based talking head models generate high-quality, photorealistic videos but suffer from slow inference.<n>We introduce Lightning-fast Caching-based Parallel denoising prediction (LightningCP)<n>We also propose Decoupled Foreground Attention (DFA) to further accelerate attention computations.
  arXiv  Detail & Related papers  (2025-08-25T02:58:39Z)
• FPSAttention: Training-Aware FP8 and Sparsity Co-Design for Fast Video Diffusion [44.206702976963676]
  We introduce FPSAttention, a novel training-aware co-design of FP8 quantization and sparsity for video generation.<n>Our approach features three key innovations: 1) A unified 3D tile-wise granularity that simultaneously supports both quantization and sparsity; 2) A denoising step-aware strategy that adapts to the noise schedule, addressing the strong correlation between quantization/sparsity errors and denoising steps; 3) A native, hardware-friendly kernel that leverages FlashAttention and is implemented with Hopper optimized architecture features for highly efficient execution.
  arXiv  Detail & Related papers  (2025-06-05T05:30:30Z)
• DiVE: Efficient Multi-View Driving Scenes Generation Based on Video Diffusion Transformer [56.98400572837792]
  DiVE produces high-fidelity, temporally coherent, and cross-view consistent multi-view videos.<n>These innovations collectively achieve a 2.62x speedup with minimal quality degradation.
  arXiv  Detail & Related papers  (2025-04-28T09:20:50Z)
• DSV: Exploiting Dynamic Sparsity to Accelerate Large-Scale Video DiT Training [85.04885553561164]
  Diffusion Transformers (DiTs) have shown remarkable performance in generating high-quality videos.<n>DiTs can consume up to 95% of processing time and demand specialized context parallelism.<n>This paper introduces DSV to accelerate video DiT training by leveraging the dynamic attention sparsity we empirically observe.
  arXiv  Detail & Related papers  (2025-02-11T14:39:59Z)
• Vchitect-2.0: Parallel Transformer for Scaling Up Video Diffusion Models [89.79067761383855]
  Vchitect-2.0 is a parallel transformer architecture designed to scale up video diffusion models for large-scale text-to-video generation.<n>By introducing a novel Multimodal Diffusion Block, our approach achieves consistent alignment between text descriptions and generated video frames.<n>To overcome memory and computational bottlenecks, we propose a Memory-efficient Training framework.
  arXiv  Detail & Related papers  (2025-01-14T21:53:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.