DFSAttn: Dynamic Fine-grained Sparse Attention for Efficient Video Generation

• URL: http://arxiv.org/abs/2605.23445v1
• Date: Fri, 22 May 2026 09:58:23 GMT
• Title: DFSAttn: Dynamic Fine-grained Sparse Attention for Efficient Video Generation
• Abstract summary: Diffusion 3D full attention incurs prohibitive computational cost due to the quadratic complexity of attention.<n>We propose DFSAttn, a training-free sparse attention framework that enables dynamic, fine-grained sparsification efficiently.
• Score: 15.509814124582926
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Diffusion transformers have achieved remarkable success in high-quality video generation, yet their reliance on spatiotemporal 3D full attention incurs prohibitive computational cost due to the quadratic complexity of attention. Block sparse attention is a common approach to mitigate this by focusing computation on important regions. However, attention maps in DiTs exhibit inherently dynamic and fine-grained sparsity, which causes existing block sparse attention methods to degrade significantly in quality, especially at high sparsity ratios. In this paper, we revisit block sparse attention and derive a theoretical lower bound on attention recall to characterize the key factors governing its effectiveness. Guided by these insights, we propose DFSAttn, a training-free sparse attention framework that enables dynamic, fine-grained sparsification efficiently. DFSAttn incorporates three core designs: Hilbert curve-based token reordering to achieve fine-grained sparsity while preserving efficient GPU execution, hierarchical block scoring for accurate block importance estimation, and sparse mask caching with adaptive ratios to balance accuracy and efficiency. Experimental results demonstrate that DFSAttn consistently outperforms prior methods under high sparsity, achieving up to 2.1$\times$ end-to-end speedup while maintaining high generation quality. Our code is open-sourced and available at https://github.com/jessica-hujie/DFSAttn.

Related papers

• Beyond Heuristics: Learnable Density Control for 3D Gaussian Splatting [78.96382179408054]
  We introduce bfLeGS, a framework that reformulates density control as a parameterized policy network optimized via Reinforcement Learning (RL)<n>To maintain computational tractability, we derive a closed-form solution that reduces the complexity of reward calculation from $O(N2)$ to $O(N)$.<n>Experiments on the Mip-NeRF 360, Tanks & Temples, and Deep Blending datasets demonstrate that bfLeGS significantly outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-05-01T05:04:49Z)
• SparVAR: Exploring Sparsity in Visual AutoRegressive Modeling for Training-Free Acceleration [23.86429472943524]
  We present a training-free acceleration framework that exploits three properties of Visual AutoRegressive attention: strong attention sinks, cross-scale activation similarity, and pronounced locality.<n>Specifically, we dynamically predict the sparse attention pattern of later high-resolution scales from a sparse decision scale, and construct scale self-similar sparse attention via an efficient index-mapping mechanism.<n>Our method achieves a $mathbf1.57times$ speed-up while preserving almost all high-frequency details.
  arXiv  Detail & Related papers  (2026-02-04T09:34:06Z)
• PISA: Piecewise Sparse Attention Is Wiser for Efficient Diffusion Transformers [37.401543107035046]
  Diffusion Transformers are fundamental for video and image generation, but their efficiency is bottlenecked by the quadratic complexity of attention.<n>We propose PISA, a training-free Piecewise Sparse Attention that covers the full attention span with sub-quadratic complexity.
  arXiv  Detail & Related papers  (2026-02-01T07:47:06Z)
• Causal Autoregressive Diffusion Language Model [70.7353007255797]
  CARD reformulates the diffusion process within a strictly causal attention mask, enabling dense, per-token supervision in a single forward pass.<n>Our results demonstrate that CARD achieves ARM-level data efficiency while unlocking the latency benefits of parallel generation.
  arXiv  Detail & Related papers  (2026-01-29T17:38:29Z)
• PSA: Pyramid Sparse Attention for Efficient Video Understanding and Generation [34.8993443618652]
  We present Pyramid Sparse Attention (PSA), a versatile module applicable to both video understanding and generation tasks.<n>Instead of binary masking, PSA introduces multi-level pooled KV representations, enabling finer mask granularity.<n>This design, analogous to fixed-point quantization and classical feature pyramid networks in computer vision, effectively mitigates information loss while preserving computational efficiency under a low compute budget.
  arXiv  Detail & Related papers  (2025-12-03T18:02:11Z)
• ProxyAttn: Guided Sparse Attention via Representative Heads [59.03412871683236]
  We propose ProxyAttn, a training-free sparse attention algorithm that achieves more precise block estimation.<n>We show that ProxyAttn can achieve up to 10.3x attention acceleration and 2.4x prefilling acceleration without significant performance loss.
  arXiv  Detail & Related papers  (2025-09-29T13:10:39Z)
• Bidirectional Sparse Attention for Faster Video Diffusion Training [14.523882232476092]
  Video diffusion Transformer (DiT) models excel in generative quality but hit major computational bottlenecks when producing high-resolution, long-duration videos.<n>We propose a Bidirectional Sparse Attention (BSA) framework for faster video DiT training, the first to dynamically sparsify both Queries and Key-Value pairs within 3D full attention.<n>BSA significantly accelerates DiT training across long sequences, reducing FLOPs by up to 20x and achieving 17.79x faster attention training, while preserving or even surpassing the generative quality of full attention.
  arXiv  Detail & Related papers  (2025-09-01T03:16:52Z)
• Delta Attention: Fast and Accurate Sparse Attention Inference by Delta Correction [52.14200610448542]
  A transformer has a quadratic complexity, leading to high inference costs and latency for long sequences.<n>We propose a simple, novel, and effective procedure for correcting this distributional shift.<n>Our method can maintain approximately 98.5% sparsity over full quadratic attention, making our model 32 times faster than Flash Attention 2 when processing 1M token prefills.
  arXiv  Detail & Related papers  (2025-05-16T13:48:33Z)
• iFlame: Interleaving Full and Linear Attention for Efficient Mesh Generation [49.8026360054331]
  iFlame is a novel transformer-based network architecture for mesh generation.<n>We propose an interleaving autoregressive mesh generation framework that combines the efficiency of linear attention with the expressive power of full attention mechanisms.<n>Our results indicate that the proposed interleaving framework effectively balances computational efficiency and generative performance.
  arXiv  Detail & Related papers  (2025-03-20T19:10:37Z)
• XAttention: Block Sparse Attention with Antidiagonal Scoring [10.517760961650279]
  Long-context Transformer Models (LCTMs) are vital for real-world applications but suffer high computational costs due to attention's quadratic complexity.<n>We introduce XAttention, a plug-and-play framework that dramatically accelerates long-context inference in Transformers models using sparse attention.
  arXiv  Detail & Related papers  (2025-03-20T17:59:58Z)
• UNETR++: Delving into Efficient and Accurate 3D Medical Image Segmentation [93.88170217725805]
  We propose a 3D medical image segmentation approach, named UNETR++, that offers both high-quality segmentation masks as well as efficiency in terms of parameters, compute cost, and inference speed. The core of our design is the introduction of a novel efficient paired attention (EPA) block that efficiently learns spatial and channel-wise discriminative features. Our evaluations on five benchmarks, Synapse, BTCV, ACDC, BRaTs, and Decathlon-Lung, reveal the effectiveness of our contributions in terms of both efficiency and accuracy.
  arXiv  Detail & Related papers  (2022-12-08T18:59:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.