HyperVAttention: Efficient Sparse Attention with Spatio-Temporal Clustering for Video Diffusion

• URL: http://arxiv.org/abs/2607.03012v1
• Date: Fri, 03 Jul 2026 06:46:35 GMT
• Title: HyperVAttention: Efficient Sparse Attention with Spatio-Temporal Clustering for Video Diffusion
• Abstract summary: Video Diffusion Transformers (VDiTs) have demonstrated significant capabilities in high-fidelity video generation.<n>Their ability to produce long-duration videos is constrained by the quadratic complexity of the self-attention mechanism.<n>Recent clustering-based sparse attention methods improve the quality-speed trade-off by grouping semantically similar tokens, but their practical efficiency remains limited by two bottlenecks.<n>We propose HyperVAttention (HVA), a training-free sparse attention framework that addresses both bottlenecks jointly.
• Score: 37.76188499190129
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Video Diffusion Transformers (VDiTs) have demonstrated significant capabilities in high-fidelity video generation. However, their ability to produce long-duration videos is fundamentally constrained by the quadratic complexity of the self-attention mechanism. Recent clustering-based sparse attention methods improve the quality-speed trade-off by grouping semantically similar tokens, but their practical efficiency remains limited by two bottlenecks: substantial clustering overhead and low CTA utilization caused by irregular cluster-induced blocks. We propose HyperVAttention (HVA), a training-free sparse attention framework that addresses both bottlenecks jointly. To reduce clustering overhead, we introduce 3D local-window clustering, which exploits the spatio-temporal locality of video tokens to restrict centroid search to fixed local neighborhoods, and implement it with a custom Triton kernel for efficient execution. We further propose a hybrid clustering strategy that performs full clustering only at anchor steps and updates only subset tokens at intermediate steps, leveraging the temporal stability of cluster assignments across denoising steps. To improve CTA utilization, we present hardware-aware cluster merging that minimizes CTA-aligned execution cost through parallel agglomerative merging, improving block density and approximation fidelity by utilizing idle tile capacity. Together, these components reduce clustering overhead, avoid redundant updates, and better align sparse attention with the fixed tile structure of modern GPU kernels. Experiments on Text-to-Video generation show that HVA establishes a new Pareto frontier for training-free sparse attention in video diffusion, reducing end-to-end latency by up to $2.13\times$ while improving fidelity over existing training-free sparse attention baselines.

Related papers

• RhymeFlow: Training-Free Acceleration for Video Generation with Asynchronous Denoising Flow Scheduling [51.279397568734424]
  Video generation models based on Diffusion Transformers (DiTs) have achieved remarkable performance in video synthesis.<n>DiTs suffer from high inference latency and computational costs due to the quadratic complexity of 3D attention.<n>We introduce bftextRhymeFlow, a training-free framework that decouples the denoising trajectories of different frames.
  arXiv  Detail & Related papers  (2026-06-04T15:49:37Z)
• AdaCluster: Adaptive Query-Key Clustering for Sparse Attention in Video Generation [9.162823040512645]
  Video diffusion transformers (DiTs) suffer from prohibitive inference latency due to quadratic attention complexity.<n>We propose AdaCluster, a training-free adaptive clustering framework.<n>Experiments on CogVideoX-2B, HunyuanVideo, and Wan-2.1 on one A40 GPU demonstrate up to 1.67-4.31x speedup with negligible quality degradation.
  arXiv  Detail & Related papers  (2026-04-20T14:43:36Z)
• Cluster-Wise Spatio-Temporal Masking for Efficient Video-Language Pretraining [86.49790441700195]
  ClusterSTM is a Cluster-Wise Spatio-Temporal Masking strategy for efficient video-language pretraining.<n>We introduce a video-text relevance reconstruction objective that aligns high-level multimodal semantics beyond conventional visual reconstruction.
  arXiv  Detail & Related papers  (2026-03-24T08:48:15Z)
• DiEC: Diffusion Embedded Clustering [0.76629754443761]
  Deep clustering critically depends on representations that expose clear cluster structure.<n>Most prior methods learn a single embedding with an autoencoder or a self-supervised encoder and treat it as the primary representation for clustering.<n>We propose Embedded Diffusion Clustering (DiEC), an unsupervised clustering framework that exploits this trajectory by directly leveraging intermediate activations of a pretrained diffusion U-Net.
  arXiv  Detail & Related papers  (2025-12-24T03:10:00Z)
• You Can Trust Your Clustering Model: A Parameter-free Self-Boosting Plug-in for Deep Clustering [73.48306836608124]
  DCBoost is a parameter-free plug-in designed to enhance the global feature structures of current deep clustering models.<n>By harnessing reliable local structural cues, our method aims to elevate clustering performance effectively.
  arXiv  Detail & Related papers  (2025-11-26T09:16:36Z)
• Compact Attention: Exploiting Structured Spatio-Temporal Sparsity for Fast Video Generation [21.87891961960399]
  Compact Attention is a hardware-aware acceleration framework featuring three innovations.<n>Our method achieves 1.62.5x acceleration in attention on single- GPU setups.<n>This work provides a principled approach to unlocking efficient long-form video generation through structured sparsity exploitation.
  arXiv  Detail & Related papers  (2025-08-18T14:45:42Z)
• Temporal Cluster Assignment for Efficient Real-Time Video Segmentation [9.248291541710781]
  Vision Transformers have substantially advanced the capabilities of segmentation models across both image and video domains.<n>The window-based attention mechanism of Swin requires a fixed number of tokens per window, limiting the applicability of conventional pruning techniques.<n>We introduce Temporal Cluster Assignment (TCA), a lightweight and effective, fine-tuning-free strategy that enhances token clustering by leveraging temporal coherence.
  arXiv  Detail & Related papers  (2025-08-07T20:52:49Z)
• Revisiting Self-Supervised Heterogeneous Graph Learning from Spectral Clustering Perspective [52.662463893268225]
  Self-supervised heterogeneous graph learning (SHGL) has shown promising potential in diverse scenarios.<n>Existing SHGL methods encounter two significant limitations.<n>We introduce a novel framework enhanced by rank and dual consistency constraints.
  arXiv  Detail & Related papers  (2024-12-01T09:33:20Z)
• Improving Weakly-supervised Video Instance Segmentation by Leveraging Spatio-temporal Consistency [9.115508086522887]
  We introduce a weakly-supervised method called Eigen VIS that achieves competitive accuracy compared to other VIS approaches. This method is based on two key innovations: a Temporal Eigenvalue Loss (TEL) and a clip-level Quality Co-efficient (QCC) The code is available on https://github.com/farnooshar/EigenVIS.
  arXiv  Detail & Related papers  (2024-08-29T16:05:05Z)
• CenterCLIP: Token Clustering for Efficient Text-Video Retrieval [67.21528544724546]
  In CLIP, the essential visual tokenization process, which produces discrete visual token sequences, generates many homogeneous tokens due to the redundancy nature of consecutive frames in videos. This significantly increases computation costs and hinders the deployment of video retrieval models in web applications. In this paper, we design a multi-segment token clustering algorithm to find the most representative tokens and drop the non-essential ones.
  arXiv  Detail & Related papers  (2022-05-02T12:02:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.