Veda: Scalable Video Diffusion via Distilled Sparse Attention

• URL: http://arxiv.org/abs/2605.30325v1
• Date: Thu, 28 May 2026 17:57:07 GMT
• Title: Veda: Scalable Video Diffusion via Distilled Sparse Attention
• Abstract summary: We show that generation quality is determined not by the sparsity ratio itself, but by how well the sparse mask aligns with the tile-wise geometry of full attention.<n>We propose Veda, a distilled sparse attention framework that formulates tile selection as an explicit reconstruction problem from full attention.
• Score: 38.39010826063852
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Scaling Diffusion Transformers to generate high-resolution, long videos is constrained by the quadratic cost of self-attention, and existing sparse attention methods degrade under high sparsity. We show empirically that generation quality is determined not by the sparsity ratio itself, but by how well the sparse mask aligns with the tile-wise geometry of full attention. Based on this insight, we propose Veda, a distilled sparse attention framework that formulates tile selection as an explicit reconstruction problem from full attention. Veda integrates statistics-aware tile scoring with head-aware tiling to reduce estimation error and structural mismatch, enabling aggressive sparsity. A hardware-efficient tile-skipping kernel converts theoretical sparsity into practical wall-clock speedups. Experiments on large video diffusion models, including Waver and Wan2.1, demonstrate substantial acceleration with no noticeable degradation in generation quality. To generate 720P 10-second videos on Waver-T2V-12B, Veda achieves a 5.1$\times$ end-to-end speedup and a 10.5$\times$ self-attention speedup, reducing attention overhead from 92% to 50%. Notably, the gains increase with sequence length, indicating that Veda scales favorably with spatiotemporal resolution across models.

Related papers

• HASTE: Training-Free Video Diffusion Acceleration via Head-Wise Adaptive Sparse Attention [68.95533683996236]
  Training-free sparse attention is attractive because it accelerates pretrained models without retraining.<n>Existing online top-$p$ sparse attention still spends non-negligible cost on mask prediction.<n>We show that these two overlooked factors limit the practical speed-quality trade-off of training-free sparse attention in Video DiTs.
  arXiv  Detail & Related papers  (2026-05-14T07:57:55Z)
• Sparse Forcing: Native Trainable Sparse Attention for Real-time Autoregressive Diffusion Video Generation [12.765068538856013]
  We introduce Sparse Forcing, a training-and-inference paradigm for autoregressive video diffusion models.<n>Sparse Forcing improves long-horizon generation quality while reducing decoding latency.
  arXiv  Detail & Related papers  (2026-04-23T02:22:25Z)
• Adaptive Spectral Feature Forecasting for Diffusion Sampling Acceleration [58.19554276924402]
  We propose spectral diffusion feature forecaster (Spectrum) to enable global, long-range feature reuse with tightly controlled error.<n>We achieve up to 4.79$times$ speedup on FLUX.1 and 4.67$times$ speedup on Wan2.1-14B, while maintaining much higher sample quality compared with the baselines.
  arXiv  Detail & Related papers  (2026-03-02T08:59:11Z)
• SALAD: Achieve High-Sparsity Attention via Efficient Linear Attention Tuning for Video Diffusion Transformer [58.79642223409644]
  Diffusion Transformers have recently demonstrated remarkable performance in video generation.<n>We propose SALAD, introducing a lightweight linear attention branch in parallel with the sparse attention.<n>Our method attains 90% sparsity and 1.72x inference speedup, while maintaining generation quality comparable to the full attention baseline.
  arXiv  Detail & Related papers  (2026-01-23T07:28:53Z)
• SLA: Beyond Sparsity in Diffusion Transformers via Fine-Tunable Sparse-Linear Attention [88.47701139980636]
  In Diffusion Transformer (DiT) models, particularly for video generation, attention latency is a major bottleneck.<n>We find that attention weights can be separated into two parts: a small fraction of large weights with high rank and the remaining weights with very low rank.<n>We propose SLA, a trainable attention method that fuses sparse and linear attention to accelerate diffusion models.
  arXiv  Detail & Related papers  (2025-09-28T17:58:59Z)
• QuantSparse: Comprehensively Compressing Video Diffusion Transformer with Model Quantization and Attention Sparsification [67.15451442018258]
  Diffusion transformers exhibit remarkable video generation capability, yet their prohibitive computational and memory costs hinder practical deployment.<n>Model quantization and attention sparsification are two promising directions for compression, but each alone suffers severe performance degradation under aggressive compression.<n>We propose textbfQuantSparse, a unified framework that integrates model quantization with attention sparsification.
  arXiv  Detail & Related papers  (2025-09-28T06:49:44Z)
• Sparse-vDiT: Unleashing the Power of Sparse Attention to Accelerate Video Diffusion Transformers [24.105473321347894]
  We propose Sparse-vDiT, a sparsity acceleration framework for Video Diffusion Transformer (vDiT)<n>We show that Sparse-vDiT achieves 2.09$times$, 2.38$times$, and 1.67$times$ theoretical FLOP reduction, and actual inference speedups of 1.76$times$, 1.85$times$, and 1.58$times$, respectively.<n>Our work demonstrates that latent structural sparsity in vDiTs can be systematically exploited for long video synthesis.
  arXiv  Detail & Related papers  (2025-06-03T16:42:37Z)
• Re-ttention: Ultra Sparse Visual Generation via Attention Statistical Reshape [38.76559841681518]
  A huge bottleneck is the attention mechanism where complexity scales quadratically with resolution and video length.<n>Existing techniques fail to preserve visual quality at extremely high sparsity levels and might even incur non-negligible compute overheads.<n>We propose Re-ttention, which implements very high sparse attention for visual generation models.
  arXiv  Detail & Related papers  (2025-05-28T22:39:12Z)
• VSA: Faster Video Diffusion with Trainable Sparse Attention [38.37291040904089]
  Scaling video diffusion transformers (DiTs) is limited by their quadratic 3D attention, even though most of the attention mass concentrates on a small subset of positions.<n>We turn this observation into VSA, a trainable, hardware-efficient sparse attention that replaces full attention at emphboth training and inference.
  arXiv  Detail & Related papers  (2025-05-19T17:30:13Z)
• DraftAttention: Fast Video Diffusion via Low-Resolution Attention Guidance [43.423240627266644]
  Diffusion transformer-based video generation models (DiTs) have recently attracted widespread attention for their excellent generation quality.<n>However, their computational cost remains a major bottleneck-attention alone accounts for over 80% of total latency.<n>We propose the DraftAttention, a training-free framework for the acceleration of video diffusion transformers with dynamic sparse attention on GPUs.
  arXiv  Detail & Related papers  (2025-05-17T04:34:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.