RoPeSLR: 3D RoPE-driven Sparse-LowRank Attention for Efficient Diffusion Transformers

• URL: http://arxiv.org/abs/2605.20659v1
• Date: Wed, 20 May 2026 03:24:50 GMT
• Title: RoPeSLR: 3D RoPE-driven Sparse-LowRank Attention for Efficient Diffusion Transformers
• Abstract summary: Diffusion Transformers (DiTs) have revolutionized high-fidelity video generation, yet their $mathcalO(L2)$ attention complexity poses a formidable bottleneck for long-sequence synthesis.<n>We propose textbfRoPe SLR, a 3D RoPE-guided Sparse-LowRank attention framework.<n>RoPe SLR achieves up to $10times$ fewer FLOPs on Wan2.1-1.3B and delivers a $2.26times$ end-to-end inference speedup.
• Score: 15.551959210208862
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Diffusion Transformers (DiTs) have revolutionized high-fidelity video generation, yet their $\mathcal{O}(L^2)$ attention complexity poses a formidable bottleneck for long-sequence synthesis. While recent sparse-linear attention hybrids aim to mitigate this, their performance severely degrades at extreme sparsity due to the "RoPE Dilemma": standard linear attention fails to preserve the orthogonal relative-position structure of 3D Rotary Position Embeddings (RoPE), neutralizing vital distance awareness. To address this, we propose \textbf{RoPeSLR}, a 3D RoPE-guided Sparse-LowRank attention framework. We establish that under empirically validated assumptions, the DiT attention manifold admits a decoupling into a high-frequency semantic spike set (bounded by $\mathcal{O}(L^{3/2})$ sparsity) and an extreme low-rank ($\mathcal{O}(d_h \log L)$) background continuum. Guided by this structural prior, RoPeSLR eschews standard linear attention for a head-wise low-rank parameterization equipped with a learnable 3D Absolute Positional Embedding (PE) injection, seamlessly synthesizing long-range relative distance decay. By guaranteeing sub-quadratic sparsity and sub-linear rank growth, RoPeSLR is exceptionally suited for scaling to ultra-long video inference. Extensive evaluations validate this scalable superiority: at 90\% sparsity, RoPeSLR achieves up to $10\times$ fewer FLOPs on Wan2.1-1.3B and delivers a $2.26\times$ end-to-end inference speedup on the ultra-long 100K+ token sequences of HunyuanVideo-13B, all while maintaining near-lossless generation fidelity (less than 1.3\% average VBench degradation).

Related papers

• RoLA: Rotary-Positioned Low-Rank Linear Attention for Efficient Diffusion Transformers [22.383650673776415]
  Sparse low-rank hybrids alleviate this cost by combining a local sparse branch with a global compressed branch.<n>We propose a rotary-positioned low-rank linear-attention branch that keeps genuine cross-token aggregation while remaining compatible with a reusable linear summary.<n>Experiments show that the resulting method remains competitive in generation quality at 90% sparsity while achieving 2.63$times$ end-to-end inference speedup on Wan2.B (720p, 81 frames measured on an NVIDIA H100 GPU)
  arXiv  Detail & Related papers  (2026-09-06T16:34:46Z)
• Chimera: Designing and Chinchilla-Scaling Hybrid Visual Diffusion Transformers [54.01042826416009]
  Visual generation requires high-resolution images, long videos, and multimodal context.<n>We introduce Chimera, a hybrid visual diffusion backbone with a principled scaling recipe.<n>We show that the dense backbone is 1.7x as compute-efficient as a matched full-attention Wan2.1 2B baseline.
  arXiv  Detail & Related papers  (2026-07-30T17:58:02Z)
• RotateAttention: RoPE-Aware Rotation and Range Rectification for INT4 Quantized Attention in Video Generation [15.82658615412336]
  We present an efficient textbfmixed-precision INT4 FlashAttention framework tailored for textbfDiT-based video generation models with 3D RoPE.<n>Experiments show that textbfRotateAttention preserves video generation quality nearly identical to full-precision baselines while achieving up to 1.68$times$ end-to-end speedup and 2.2$times$ kernel-level acceleration.
  arXiv  Detail & Related papers  (2026-07-01T02:44:10Z)
• Scaling Attention via Feature Sparsity [50.64995497733461]
  Scaling Transformers to ultra-long contexts is bottlenecked by the $O(n2 d)$ cost of self-attention.<n>We propose Sparse Feature Attention, where queries and keys are represented as $k$-sparse codes that preserve high-dimensional expressivity.<n>Across GPT-2 and Qwen3 pretraining, SFA matches dense baselines while improving speed by up to $2.5times$ and reducing FLOPs and KV-cache by nearly 50%.
  arXiv  Detail & Related papers  (2026-03-17T08:41:50Z)
• VSPrefill: Vertical-Slash Sparse Attention with Lightweight Indexing for Long-Context Prefilling [0.0]
  Existing sparse attention methods face a trade-off among context adaptivity, sampling overhead, and fine-tuning costs.<n>We propose VSPrefill, a mechanism requiring lightweight training that uses the vertical-slash structural pattern in attention distributions.<n>VSPrefill preserves 98.35% of the full attention accuracy while delivering a 4.95x average speedup at a context length of 128k.
  arXiv  Detail & Related papers  (2026-03-03T09:24:58Z)
• Infinity-RoPE: Action-Controllable Infinite Video Generation Emerges From Autoregressive Self-Rollout [15.899488263212442]
  $infty$-RoPE is a unified inference-time framework for autoregressive video diffusion.<n>Block-Relativistic RoPE reformulates temporal encoding as a moving local reference frame.<n> KV Flush renews the KV cache by retaining only two latent frames, the global sink and the last generated latent frame.<n>RoPE Cut introduces controlled discontinuities in temporal RoPE coordinates.
  arXiv  Detail & Related papers  (2025-11-25T18:59:46Z)
• PAS: A Training-Free Stabilizer for Temporal Encoding in Video LLMs [57.790910044227935]
  Video LLMs suffer from temporal inconsistency: small shifts in frame timing can flip attention and suppress relevant frames.<n>We present Phase Aggregated Smoothing (PAS), a training-free mechanism that applies small opposed phase offsets across heads and then aggregates their outputs.<n>Our analysis shows that the RoPE rotated logit can be approximated as a content dot product scaled by a time kernel; smoothing this kernel yields Lipschitz stability of attention to small temporal shifts; multi phase averaging attenuates high frequency ripples while preserving per-head spectra under Nyquist-valid sampling.
  arXiv  Detail & Related papers  (2025-11-14T05:56:47Z)
• DoPE: Denoising Rotary Position Embedding [60.779039511252584]
  Rotary Position Embedding (RoPE) in Transformer models has inherent limits that weaken length.<n>We reinterpret the attention map with positional encoding as a noisy feature map, and propose Denoising Positional extrapolation page (DoPE)<n>DoPE is a training-free method based on truncated matrix entropy to detect outlier frequency bands in the feature map.
  arXiv  Detail & Related papers  (2025-11-12T09:32:35Z)
• SLA: Beyond Sparsity in Diffusion Transformers via Fine-Tunable Sparse-Linear Attention [88.47701139980636]
  In Diffusion Transformer (DiT) models, particularly for video generation, attention latency is a major bottleneck.<n>We find that attention weights can be separated into two parts: a small fraction of large weights with high rank and the remaining weights with very low rank.<n>We propose SLA, a trainable attention method that fuses sparse and linear attention to accelerate diffusion models.
  arXiv  Detail & Related papers  (2025-09-28T17:58:59Z)
• Lumos-1: On Autoregressive Video Generation from a Unified Model Perspective [37.58855048653859]
  Autoregressive large language models (LLMs) have unified a vast range of language tasks, inspiring preliminary efforts of autoregressive video generation.<n>Lumos-1 retains autoregressive video generator architecture with minimal architectural modifications.
  arXiv  Detail & Related papers  (2025-07-11T17:59:42Z)
• Sparse-vDiT: Unleashing the Power of Sparse Attention to Accelerate Video Diffusion Transformers [24.105473321347894]
  We propose Sparse-vDiT, a sparsity acceleration framework for Video Diffusion Transformer (vDiT)<n>We show that Sparse-vDiT achieves 2.09$times$, 2.38$times$, and 1.67$times$ theoretical FLOP reduction, and actual inference speedups of 1.76$times$, 1.85$times$, and 1.58$times$, respectively.<n>Our work demonstrates that latent structural sparsity in vDiTs can be systematically exploited for long video synthesis.
  arXiv  Detail & Related papers  (2025-06-03T16:42:37Z)
• VSA: Faster Video Diffusion with Trainable Sparse Attention [38.37291040904089]
  Scaling video diffusion transformers (DiTs) is limited by their quadratic 3D attention, even though most of the attention mass concentrates on a small subset of positions.<n>We turn this observation into VSA, a trainable, hardware-efficient sparse attention that replaces full attention at emphboth training and inference.
  arXiv  Detail & Related papers  (2025-05-19T17:30:13Z)
• Rotation-Invariant Transformer for Point Cloud Matching [42.5714375149213]
  We introduce RoITr, a Rotation-Invariant Transformer to cope with the pose variations in the point cloud matching task. We propose a global transformer with rotation-invariant cross-frame spatial awareness learned by the self-attention mechanism. RoITr surpasses the existing methods by at least 13 and 5 percentage points in terms of Inlier Ratio and Registration Recall.
  arXiv  Detail & Related papers  (2023-03-14T20:55:27Z)
• Pyramid R-CNN: Towards Better Performance and Adaptability for 3D Object Detection [89.66162518035144]
  We present a flexible and high-performance framework, named Pyramid R-CNN, for two-stage 3D object detection from point clouds. We propose a novel second-stage module, named pyramid RoI head, to adaptively learn the features from the sparse points of interest. Our pyramid RoI head is robust to the sparse and imbalanced circumstances, and can be applied upon various 3D backbones to consistently boost the detection performance.
  arXiv  Detail & Related papers  (2021-09-06T14:17:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.