AdaCluster: Adaptive Query-Key Clustering for Sparse Attention in Video Generation

• URL: http://arxiv.org/abs/2604.18348v1
• Date: Mon, 20 Apr 2026 14:43:36 GMT
• Title: AdaCluster: Adaptive Query-Key Clustering for Sparse Attention in Video Generation
• Abstract summary: Video diffusion transformers (DiTs) suffer from prohibitive inference latency due to quadratic attention complexity.<n>We propose AdaCluster, a training-free adaptive clustering framework.<n>Experiments on CogVideoX-2B, HunyuanVideo, and Wan-2.1 on one A40 GPU demonstrate up to 1.67-4.31x speedup with negligible quality degradation.
• Score: 9.162823040512645
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Video diffusion transformers (DiTs) suffer from prohibitive inference latency due to quadratic attention complexity. Existing sparse attention methods either overlook semantic similarity or fail to adapt to heterogeneous token distributions across layers, leading to model performance degradation. We propose AdaCluster, a training-free adaptive clustering framework that accelerates the generation of DiTs while preserving accuracy. AdaCluster applies an angle-similarity-preserving clustering method to query vectors for higher compression, and designs a euclidean-similarity-preserving clustering method for keys, covering cluster number assignment, threshold-wise adaptive clustering, and efficient critical cluster selection. Experiments on CogVideoX-2B, HunyuanVideo, and Wan-2.1 on one A40 GPU demonstrate up to 1.67-4.31x speedup with negligible quality degradation.

Related papers

• ClusterAttention: A training-free speedup of bidirectional attention [0.0]
  ClusterAttention is a training-free speedup of bidirectional attention layers.<n>On large-scale data ClusterAttention speeds up TabPFN-3 arXiv:2605.13986 by two to six times.
  arXiv  Detail & Related papers  (2026-08-27T11:04:41Z)
• HyperVAttention: Efficient Sparse Attention with Spatio-Temporal Clustering for Video Diffusion [37.76188499190129]
  Video Diffusion Transformers (VDiTs) have demonstrated significant capabilities in high-fidelity video generation.<n>Their ability to produce long-duration videos is constrained by the quadratic complexity of the self-attention mechanism.<n>Recent clustering-based sparse attention methods improve the quality-speed trade-off by grouping semantically similar tokens, but their practical efficiency remains limited by two bottlenecks.<n>We propose HyperVAttention (HVA), a training-free sparse attention framework that addresses both bottlenecks jointly.
  arXiv  Detail & Related papers  (2026-07-03T06:46:35Z)
• DiEC: Diffusion Embedded Clustering [0.76629754443761]
  Deep clustering critically depends on representations that expose clear cluster structure.<n>Most prior methods learn a single embedding with an autoencoder or a self-supervised encoder and treat it as the primary representation for clustering.<n>We propose Embedded Diffusion Clustering (DiEC), an unsupervised clustering framework that exploits this trajectory by directly leveraging intermediate activations of a pretrained diffusion U-Net.
  arXiv  Detail & Related papers  (2025-12-24T03:10:00Z)
• You Can Trust Your Clustering Model: A Parameter-free Self-Boosting Plug-in for Deep Clustering [73.48306836608124]
  DCBoost is a parameter-free plug-in designed to enhance the global feature structures of current deep clustering models.<n>By harnessing reliable local structural cues, our method aims to elevate clustering performance effectively.
  arXiv  Detail & Related papers  (2025-11-26T09:16:36Z)
• Self-Enhanced Image Clustering with Cross-Modal Semantic Consistency [57.961869351897384]
  We propose a framework based on cross-modal semantic consistency for efficient image clustering.<n>Our framework first builds a strong foundation via Cross-Modal Semantic Consistency.<n>In the first stage, we train lightweight clustering heads to align with the rich semantics of the pre-trained model.<n>In the second stage, we introduce a Self-Enhanced fine-tuning strategy.
  arXiv  Detail & Related papers  (2025-08-02T08:12:57Z)
• Towards Learnable Anchor for Deep Multi-View Clustering [49.767879678193005]
  In this paper, we propose the Deep Multi-view Anchor Clustering (DMAC) model that performs clustering in linear time.<n>With the optimal anchors, the full sample graph is calculated to derive a discriminative embedding for clustering.<n>Experiments on several datasets demonstrate superior performance and efficiency of DMAC compared to state-of-the-art competitors.
  arXiv  Detail & Related papers  (2025-03-16T09:38:11Z)
• Self-Supervised Graph Embedding Clustering [70.36328717683297]
  K-means one-step dimensionality reduction clustering method has made some progress in addressing the curse of dimensionality in clustering tasks. We propose a unified framework that integrates manifold learning with K-means, resulting in the self-supervised graph embedding framework.
  arXiv  Detail & Related papers  (2024-09-24T08:59:51Z)
• Deep Embedding Clustering Driven by Sample Stability [16.53706617383543]
  We propose a deep embedding clustering algorithm driven by sample stability (DECS) Specifically, we start by constructing the initial feature space with an autoencoder and then learn the cluster-oriented embedding feature constrained by sample stability. The experimental results on five datasets illustrate that the proposed method achieves superior performance compared to state-of-the-art clustering approaches.
  arXiv  Detail & Related papers  (2024-01-29T09:19:49Z)
• Stable Cluster Discrimination for Deep Clustering [7.175082696240088]
  Deep clustering can optimize representations of instances (i.e., representation learning) and explore the inherent data distribution. The coupled objective implies a trivial solution that all instances collapse to the uniform features. In this work, we first show that the prevalent discrimination task in supervised learning is unstable for one-stage clustering. A novel stable cluster discrimination (SeCu) task is proposed and a new hardness-aware clustering criterion can be obtained accordingly.
  arXiv  Detail & Related papers  (2023-11-24T06:43:26Z)
• Enhancing Latent Space Clustering in Multi-filter Seq2Seq Model: A Reinforcement Learning Approach [0.0]
  We design a latent-enhanced multi-filter seq2seq model (LMS2S) that analyzes the latent space representations using a clustering algorithm. Our experiments on semantic parsing and machine translation demonstrate the positive correlation between the clustering quality and the model's performance.
  arXiv  Detail & Related papers  (2021-09-25T16:36:31Z)
• Scalable Hierarchical Agglomerative Clustering [65.66407726145619]
  Existing scalable hierarchical clustering methods sacrifice quality for speed. We present a scalable, agglomerative method for hierarchical clustering that does not sacrifice quality and scales to billions of data points.
  arXiv  Detail & Related papers  (2020-10-22T15:58:35Z)
• Contrastive Clustering [57.71729650297379]
  We propose Contrastive Clustering (CC) which explicitly performs the instance- and cluster-level contrastive learning. In particular, CC achieves an NMI of 0.705 (0.431) on the CIFAR-10 (CIFAR-100) dataset, which is an up to 19% (39%) performance improvement compared with the best baseline.
  arXiv  Detail & Related papers  (2020-09-21T08:54:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.