SQuad: Sub-Quadratic Attention Distillation for Efficient Video Generation

• URL: http://arxiv.org/abs/2608.16585v1
• Date: Mon, 17 Aug 2026 13:47:44 GMT
• Title: SQuad: Sub-Quadratic Attention Distillation for Efficient Video Generation
• Abstract summary: Video Diffusion Transformers (DiTs) spend most of their compute inside the Self-Attention operation.<n>We propose SQuad, a Sub-Quadratic Attention Distillation framework that achieves a complexity of $mathcalO(nsqrtn)$ in the resulting distilled Attention.
• Score: 14.732232216463814
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Video Diffusion Transformers (DiTs) spend most of their compute inside the Self-Attention operation, whose cost grows quadratically, $\mathcal{O}(n^2)$, with the number of latent tokens $n$. For the task of video generation, the token count is large, so this term dominates runtime and memory, and thereby caps the resolution and duration we can generate. Linear $\mathcal{O}(n)$ and low-rank $\mathcal{O}(nk)$ surrogates of Self-Attention trade the full softmax $QK^T$ for cheaper kernels, but rarely recover the original's expressivity, leaving a stubborn quality gap. Motivated by this, we propose SQuad, a Sub-Quadratic Attention Distillation framework that achieves a complexity of $\mathcal{O}(n\sqrt{n})$ in the resulting distilled Attention, naturally balancing the efficiency v/s expressivity trade-off. Instead of training our own Video DiT from scratch, which is prohibitively expensive, we fit a pretrained full softmax Self-Attention DiT into our proposed SQuad-Attention one by distilling the former in two stages: Flow-Matching Supervised Fine-Tuning (SFT), followed by improved Distribution Matching Distillation (DMD2) which additionally makes the sampling more efficient. On the Wan~2.2 5B text-to-video model, SQuAD matches the quadratic teacher on VBench ($83.20$ v/s $83.08$) while cutting the per-step per-block attention FLOPs by $\sim$$67\times$ and attention latency by $\sim$$11\times$, and end-to-end DiT latency by 2$\times$, all while also generating a video in only $6$ Neural Functional Evaluations (NFEs) instead of the default $100$.

Related papers

• Asymptotically Optimal Regret for Reinforcement Learning without Horizon Dependence [54.59847568544922]
  We study horizon-free regret for finite-horizon time-homogeneous Markov decision processes with $A$ states, $A$ actions, horizon $H$, and per-trajectory total reward bounded by $1$.<n>We prove a regret upper bound [tilde O(sqrtSAK+S8A3)] with failure probability $$, where $K$ is the number of episodes and $tilde O(sqrtSAK+S3K)$ hides $mathsfpoly
  arXiv  Detail & Related papers  (2026-07-22T07:42:19Z)
• Kernelized Linear Attention: Breaking the Capacity Wall with Symmetric Cones [4.847095054351615]
  Linear attention promises constant-time recurrent inference but degrades sharply on associative recall.<n>We introduce Kernelized Linear Attention Activations (KATA), a framework whose feature maps are derived from first principles.<n>We show that rank-one positive semi-definite features offer a favorable capacity--interference tradeoff.
  arXiv  Detail & Related papers  (2026-07-19T21:46:09Z)
• Closing the Theory-Practice Gap in Spiking Transformers via Effective Dimension [13.891522069967507]
  Spiking transformers achieve competitive accuracy with conventional transformers while offering $38$-$57times$ energy efficiency on neuromorphic hardware.<n>This paper establishes the first comprehensive expressivity theory for spiking self-attention.
  arXiv  Detail & Related papers  (2026-04-17T07:15:53Z)
• Transform Trained Transformer: Accelerating Naive 4K Video Generation Over 10$\times$ [91.61519033897424]
  Native 4K video generation remains a critical challenge due to the quadratic computational explosion of full-attention as resolution increases.<n>This paper proposes a novel Transformer retrofit strategy termed $textbfT3-Video$ that significantly reduces compute requirements by optimizing their forward logic.<n>Results on 4K-VBench show that $textbfT3-Video$ substantially outperforms existing approaches.
  arXiv  Detail & Related papers  (2025-12-15T16:25:39Z)
• Test-Time Temporal Sampling for Efficient MLLM Video Understanding [26.144261085897863]
  Test-Time Temporal Sampling (T3S) is a training-free, plug-and-play inference wrapper that enables MLLMs to process long videos both efficiently and effectively.<n>Our approach operates entirely at inference time, requires no model modifications or fine-tuning, and is compatible with a wide range of pretrained MLLMs.
  arXiv  Detail & Related papers  (2025-11-22T06:59:21Z)
• H$_{2}$OT: Hierarchical Hourglass Tokenizer for Efficient Video Pose Transformers [124.11648300910444]
  We present a hierarchical plug-and-play pruning-and-$-recovering framework, called Hierarchical Hourglass Tokenizer (H$_2$OT)<n>Our method is general-purpose: it can be easily incorporated into common VPT models on both seq2seq and seq2frame pipelines.
  arXiv  Detail & Related papers  (2025-09-08T17:59:59Z)
• S$^2$Q-VDiT: Accurate Quantized Video Diffusion Transformer with Salient Data and Sparse Token Distillation [55.35880044416441]
  We propose S$2$Q-VDiT, a post-training quantization framework for video diffusion models (V-DMs)<n>Under W4A6 quantization, S$2$Q-VDiT achieves lossless performance while delivering $3.9times$ model compression and $1.3times$ inference acceleration.
  arXiv  Detail & Related papers  (2025-08-06T02:12:29Z)
• Sparse-vDiT: Unleashing the Power of Sparse Attention to Accelerate Video Diffusion Transformers [24.105473321347894]
  We propose Sparse-vDiT, a sparsity acceleration framework for Video Diffusion Transformer (vDiT)<n>We show that Sparse-vDiT achieves 2.09$times$, 2.38$times$, and 1.67$times$ theoretical FLOP reduction, and actual inference speedups of 1.76$times$, 1.85$times$, and 1.58$times$, respectively.<n>Our work demonstrates that latent structural sparsity in vDiTs can be systematically exploited for long video synthesis.
  arXiv  Detail & Related papers  (2025-06-03T16:42:37Z)
• Scaling Up Liquid-Resistance Liquid-Capacitance Networks for Efficient Sequence Modeling [50.994194925685434]
  LrcSSM is a $textitnon-linear$ recurrent model that processes long sequences as fast as today's linear state-space layers.<n>By forcing its Jacobian matrix to be diagonal, the full sequence can be solved in parallel.<n>LrcSSM offers a formal gradient-stability guarantee that other input-varying systems such as Liquid-S4 do not provide.
  arXiv  Detail & Related papers  (2025-05-27T20:02:59Z)
• AdaTP: Attention-Debiased Token Pruning for Video Large Language Models [73.70241811066907]
  Video Large Language Models (Video LLMs) have achieved remarkable results in video understanding tasks.<n>They often suffer from heavy computational overhead due to the large number of visual tokens generated from multiple video frames.<n>AdaTP integrates two dedicated debiasing modules into the pipeline, targeting global attention bias and local attention bias.
  arXiv  Detail & Related papers  (2025-05-26T15:08:37Z)
• MonarchAttention: Zero-Shot Conversion to Fast, Hardware-Aware Structured Attention [10.244490009712466]
  We propose a novel approach to sub-quadratic attention approximation via Monarch matrices.<n>MonarchAttention is both transferable, yielding minimal performance loss with no additional training, and hardware-efficient.<n>We demonstrate the quality of MonarchAttention on diverse tasks and architectures in vision and language problems.
  arXiv  Detail & Related papers  (2025-05-24T13:44:44Z)
• DiG: Scalable and Efficient Diffusion Models with Gated Linear Attention [82.24166963631949]
  Diffusion Gated Linear Attention Transformers (DiG) is a simple, adoptable solution with minimal parameter overhead.<n>We offer two variants, i,e, a plain and U-shape architecture, showing superior efficiency and competitive effectiveness.
  arXiv  Detail & Related papers  (2024-05-28T17:59:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.