ScalingAttention: Discovering Intrinsic Sparse Attention Topology for Video Diffusion Transformers

• URL: http://arxiv.org/abs/2606.23019v1
• Date: Mon, 22 Jun 2026 08:32:07 GMT
• Title: ScalingAttention: Discovering Intrinsic Sparse Attention Topology for Video Diffusion Transformers
• Abstract summary: Diffusion Transformers (DiTs) have revolutionized high-fidelity video generation, their reliance on 3D full attention creates a quadratic computational bottleneck.<n>Existing sparse methods face a dilemma: dynamic pruning suffers from prohibitive runtime overhead and memory fragmentation, while statics fail to capture fine-grained dependencies.<n>In this work, we propose ScalingAttention, a training-free framework grounded in a key inductive bias.<n> Experiments on Wan2.1 show up to 1.90X end-to-end speedup with superior fidelity, establishing a new frontier over state-of-the-art baselines.
• Score: 7.879917783645986
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While Diffusion Transformers (DiTs) have revolutionized high-fidelity video generation, their reliance on 3D full attention creates a quadratic computational bottleneck. Existing sparse methods face a dilemma: dynamic pruning suffers from prohibitive runtime overhead and memory fragmentation, while static heuristics fail to capture fine-grained dependencies. In this work, we propose ScalingAttention, a training-free framework grounded in a key inductive bias: while individual activations are input-dependent, the high-mass attention regions for each head rapidly converge to a stable, prompt-agnostic Intrinsic Sparse Topology. This topology is weight-encoded, scale-invariant, and efficient to extract. ScalingAttention decouples topology discovery from sparsity control via: (1) WEST (Weight-Encoded Sparse Topology), which extracts a robust block-sparse prior mask offline to eliminate runtime search; (2) FAST (Fidelity-Aware Sensitivity Tuning), which adaptively tunes head-wise sparsity based on diffusion fidelity requirements. To ensure practical acceleration, we co-design a hardware-aligned bit-wise block-sparse kernel. Experiments on Wan2.1 show up to 1.90X end-to-end speedup with superior fidelity, establishing a new Pareto frontier over state-of-the-art baselines.

Related papers

• Temporal Backtracking Search for Test-time Generative Video Reasoning [51.4821857044108]
  generative video reasoning remains bottlenecked by a single-shot paradigm.<n>We introduce Temporal Backtracking Search (TBS), which shifts the search space to the temporal axis.<n>In a strict out-of-distribution setting, TBS achieves 22.7%, with every solved episode stemming from a restarted branch.
  arXiv  Detail & Related papers  (2026-06-11T19:41:18Z)
• Test-time Sparsity for Extreme Fast Action Diffusion [15.679362579177267]
  Action diffusion excels at high-fidelity action generation but incurs heavy computational costs owing to its iterative denoising nature.<n>We propose test-time sparsity to tackle this challenge, which aims to accelerate action diffusion by dynamically predicting prunable residual computations for each model forward at test time.
  arXiv  Detail & Related papers  (2026-05-13T10:28:50Z)
• Ride the Wave: Precision-Allocated Sparse Attention for Smooth Video Generation [9.132288507881592]
  Video Diffusion Transformers have revolutionized high-fidelity video generation but suffer from the massive computational burden of self-attention.<n>We propose Precision-Allocated Sparse Attention (PASA), a training-free framework designed for highly efficient and temporally smooth video generation.
  arXiv  Detail & Related papers  (2026-04-14T02:51:52Z)
• Bridging Semantic and Kinematic Conditions with Diffusion-based Discrete Motion Tokenizer [55.9892973179428]
  MoTok is a discrete motion tokenizer that decouples semantic abstraction from fine-grained reconstruction.<n>On HumanML3D, our method significantly improves controllability and fidelity over MaskControl while using only one-sixth of the tokens.
  arXiv  Detail & Related papers  (2026-03-19T17:59:51Z)
• Fast-SAM3D: 3Dfy Anything in Images but Faster [65.17322167628367]
  SAM3D enables scalable, open-world 3D reconstruction from complex scenes, yet its deployment is hindered by prohibitive inference latency.<n>We present textbfFast-SAM3D, a training-free framework that aligns computation with instantaneous generation complexity.
  arXiv  Detail & Related papers  (2026-02-05T04:27:59Z)
• Compact Attention: Exploiting Structured Spatio-Temporal Sparsity for Fast Video Generation [21.87891961960399]
  Compact Attention is a hardware-aware acceleration framework featuring three innovations.<n>Our method achieves 1.62.5x acceleration in attention on single- GPU setups.<n>This work provides a principled approach to unlocking efficient long-form video generation through structured sparsity exploitation.
  arXiv  Detail & Related papers  (2025-08-18T14:45:42Z)
• Sparse-vDiT: Unleashing the Power of Sparse Attention to Accelerate Video Diffusion Transformers [24.105473321347894]
  We propose Sparse-vDiT, a sparsity acceleration framework for Video Diffusion Transformer (vDiT)<n>We show that Sparse-vDiT achieves 2.09$times$, 2.38$times$, and 1.67$times$ theoretical FLOP reduction, and actual inference speedups of 1.76$times$, 1.85$times$, and 1.58$times$, respectively.<n>Our work demonstrates that latent structural sparsity in vDiTs can be systematically exploited for long video synthesis.
  arXiv  Detail & Related papers  (2025-06-03T16:42:37Z)
• FUSE: Label-Free Image-Event Joint Monocular Depth Estimation via Frequency-Decoupled Alignment and Degradation-Robust Fusion [92.4205087439928]
  Image-event joint depth estimation methods leverage complementary modalities for robust perception, yet face challenges in generalizability.<n>We propose the Self-supervised Transfer (PST) and the FrequencyDe-coupled Fusion module (FreDF)<n>PST establishes cross-modal knowledge transfer through latent space alignment with image foundation models, effectively mitigating data scarcity.<n>FreDF explicitly decouples high-frequency edge features from low-frequency structural components, resolving modality-specific frequency mismatches.<n>This combined approach enables FUSE to construct a universal image-event that only requires lightweight decoder adaptation for target datasets.
  arXiv  Detail & Related papers  (2025-03-25T15:04:53Z)
• Training-free and Adaptive Sparse Attention for Efficient Long Video Generation [31.615453637053793]
  generating high-fidelity long videos with Diffusion Transformers (DiTs) is often hindered by significant latency.<n>We propose AdaSpa, the first Dynamic Pattern and Online Precise Search sparse attention method.<n>AdaSpa is implemented as an adaptive, plug-and-play solution and can be integrated seamlessly with existing DiTs.
  arXiv  Detail & Related papers  (2025-02-28T14:11:20Z)
• Towards Stabilized and Efficient Diffusion Transformers through Long-Skip-Connections with Spectral Constraints [51.83081671798784]
  Diffusion Transformers (DiT) have emerged as a powerful architecture for image and video generation, offering superior quality and scalability.<n>DiT's practical application suffers from inherent dynamic feature instability, leading to error amplification during cached inference.<n>We propose Skip-DiT, an image and video generative DiT variant enhanced with Long-Skip-Connections (LSCs) - the key efficiency component in U-Nets.
  arXiv  Detail & Related papers  (2024-11-26T17:28:10Z)
• Perceptron Synthesis Network: Rethinking the Action Scale Variances in Videos [48.57686258913474]
  Video action recognition has been partially addressed by the CNNs stacking of fixed-size 3D kernels. We propose to learn the optimal-scale kernels from the data. An textitaction perceptron synthesizer is proposed to generate the kernels from a bag of fixed-size kernels.
  arXiv  Detail & Related papers  (2020-07-22T14:22:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.