RotateAttention: RoPE-Aware Rotation and Range Rectification for INT4 Quantized Attention in Video Generation

• URL: http://arxiv.org/abs/2607.02584v1
• Date: Wed, 01 Jul 2026 02:44:10 GMT
• Title: RotateAttention: RoPE-Aware Rotation and Range Rectification for INT4 Quantized Attention in Video Generation
• Abstract summary: We present an efficient textbfmixed-precision INT4 FlashAttention framework tailored for textbfDiT-based video generation models with 3D RoPE.<n>Experiments show that textbfRotateAttention preserves video generation quality nearly identical to full-precision baselines while achieving up to 1.68$times$ end-to-end speedup and 2.2$times$ kernel-level acceleration.
• Score: 15.82658615412336
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In \textbf{DiT-based video generation models equipped with 3D Rotary Position Embeddings (3D RoPE)}, the attention mechanism remains a primary computational bottleneck due to its quadratic complexity with respect to sequence length. While quantized \textbf{FlashAttention} offers a promising path toward hardware acceleration, existing low-bit quantization methods overlook two critical challenges in this setting: \textbf{1)} applying online rotation matrices -- a widely used technique for mitigating outliers in Queries ($Q$) and Keys ($K$) -- is difficult to reconcile with \textbf{RoPE}; and \textbf{2)} the non-negative attention matrix $P = \exp(QK - \max(QK))$ makes symmetric quantization waste half of the 4-bit dynamic range. In this work, we observe that the outlier distributions of $Q$ and $K$ are strongly affected by the dimensional partitioning of \textbf{3D RoPE}. Based on this finding, we propose \textbf{RotateAttention}, an efficient \textbf{mixed-precision INT4 FlashAttention} framework tailored for \textbf{DiT-based video generation models with 3D RoPE}, using selective \textbf{FP16 fallback} for accuracy-sensitive attention blocks and denoising steps. RotateAttention introduces two core techniques: \textbf{1) RoPE-aware Rotation}, which employs either mergeable rotation matrices that can be fused into RoPE or negligible-overhead matrices to mitigate RoPE-induced outliers in $Q$ and $K$; and \textbf{2) Range-optimized $P$ Quantization}, which uses fixed scales and zero-points to fully exploit the \textbf{INT4 numerical range} with minimal computational overhead. Experiments show that \textbf{RotateAttention} preserves video generation quality nearly identical to full-precision baselines while achieving up to 1.68$\times$ end-to-end speedup and 2.2$\times$ kernel-level acceleration.

Related papers

• Fingerprint, Not Blueprint: How Positional Schemes Set the Default Spectral Algebra of Attention [1.7042264000899534]
  We show that the strongest previous-token heads are spectrally rotational under RoPE.<n>We also show that over public Pythia checkpoints every head originates at the random-matrix (Ginibre) null.<n>Within the settings examined, the positional scheme sets the default spectral algebra of an attention head's solution.
  arXiv  Detail & Related papers  (2026-07-07T10:17:30Z)
• Towards Fast and Effective Long Video Understanding of Multimodal Large Language Models via Adaptive Quasi-Gaussian Sampling [71.62893745623526]
  Video frame selection is a problem of Quasi-Gaussian Sampling.<n>We propose an adaptive and training-free approach termed AdaQ.
  arXiv  Detail & Related papers  (2026-06-23T06:13:30Z)
• LimiX-2M: Mitigating Low-Rank Collapse and Attention Bottlenecks in Tabular Foundation Models [56.999481798138625]
  LimiX-2M is a 2M- parameter model that outperforms larger TabPFN-v2 and TabICL baselines on widely used benchmarks.<n>We present a unified tokenize-and-route framework for strong Tabular foundation models (TFMs)<n>Results highlight value-aware tokenization and readout-aligned routing as key levers for improving the accuracy--efficiency trade-off in TFMs.
  arXiv  Detail & Related papers  (2026-06-03T06:07:33Z)
• RoPeSLR: 3D RoPE-driven Sparse-LowRank Attention for Efficient Diffusion Transformers [15.551959210208862]
  Diffusion Transformers (DiTs) have revolutionized high-fidelity video generation, yet their $mathcalO(L2)$ attention complexity poses a formidable bottleneck for long-sequence synthesis.<n>We propose textbfRoPe SLR, a 3D RoPE-guided Sparse-LowRank attention framework.<n>RoPe SLR achieves up to $10times$ fewer FLOPs on Wan2.1-1.3B and delivers a $2.26times$ end-to-end inference speedup.
  arXiv  Detail & Related papers  (2026-05-20T03:24:50Z)
• DuQuant++: Fine-grained Rotation Enhances Microscaling FP4 Quantization [47.19478866645546]
  We propose DuQuant++, which adapts the outlier-aware fine-grained rotation of DuQuant to the MXFP4 format.<n>Experiments on the LLaMA-3 family under MXFP4 W4A4 quantization show that DuQuant++ consistently achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-04-20T04:27:28Z)
• Efficient Matrix Implementation for Rotary Position Embedding [50.45885099010382]
  Rotary Position Embedding (RoPE) has become a core component of modern Transformer architectures across language, vision, and 3D domains.<n>We propose RoME, a mathematically equivalent yet computationally efficient reformulation of RoPE that replaces vector operations with unified matrix transformations.<n> experiments show that RoME delivers substantial acceleration at both the operator and full-model levels.
  arXiv  Detail & Related papers  (2026-04-10T00:17:47Z)
• Scaling Attention via Feature Sparsity [50.64995497733461]
  Scaling Transformers to ultra-long contexts is bottlenecked by the $O(n2 d)$ cost of self-attention.<n>We propose Sparse Feature Attention, where queries and keys are represented as $k$-sparse codes that preserve high-dimensional expressivity.<n>Across GPT-2 and Qwen3 pretraining, SFA matches dense baselines while improving speed by up to $2.5times$ and reducing FLOPs and KV-cache by nearly 50%.
  arXiv  Detail & Related papers  (2026-03-17T08:41:50Z)
• Rank-Aware Spectral Bounds on Attention Logits for Stable Low-Precision Training [0.0]
  Attention scores in transformers are bilinear forms $S_ij = x_itop M x_j / sqrtd_h$ whose maximum magnitude governs overflow risk in low-precision training.<n>We derive a emphrank-aware concentration inequality: when the interaction matrix $M = WQ WKtop$ has rank $r ll d$, tail probabilities for $max_i,j|S_ij|$ decay as $exp(-d22/(
  arXiv  Detail & Related papers  (2026-02-21T14:29:22Z)
• E2Former-V2: On-the-Fly Equivariant Attention with Linear Activation Memory [13.451231889715542]
  Equivariant Graph Neural Networks (EGNNs) have become a widely used approach for modeling 3D atomistic systems.<n>We introduce textbfE2Former-V2, a scalable architecture that integrates algebraic sparsity with hardware-aware execution.<n>E2Former-V2 maintains comparable predictive performance while notably accelerating inference.
  arXiv  Detail & Related papers  (2026-01-23T10:20:08Z)
• Selective Rotary Position Embedding [84.22998043041198]
  We introduce textitSelective RoPE, an textitinput-dependent rotary embedding mechanism.<n>We show that softmax attention already performs a hidden form of these rotations on query-key pairs.<n>We validate our method by equipping gated transformers with textitSelective RoPE, demonstrating that its input-dependent rotations improve performance in language modeling.
  arXiv  Detail & Related papers  (2025-11-21T16:50:00Z)
• DoPE: Denoising Rotary Position Embedding [60.779039511252584]
  Rotary Position Embedding (RoPE) in Transformer models has inherent limits that weaken length.<n>We reinterpret the attention map with positional encoding as a noisy feature map, and propose Denoising Positional extrapolation page (DoPE)<n>DoPE is a training-free method based on truncated matrix entropy to detect outlier frequency bands in the feature map.
  arXiv  Detail & Related papers  (2025-11-12T09:32:35Z)
• A3 : an Analytical Low-Rank Approximation Framework for Attention [14.649496050074735]
  We propose $tt Attt 3$, a post-training low-rank approximation framework.<n>We show that $tt Attt 3$ maintains superior performance compared to SoTAs.<n>We also demonstrate the versatility of $tt Att 3$, including KV cache compression, quantization, and mixed-rank assignments for enhanced performance.
  arXiv  Detail & Related papers  (2025-05-19T10:29:32Z)
• Gravity-aligned Rotation Averaging with Circular Regression [53.81374943525774]
  We introduce a principled approach that integrates gravity direction into the rotation averaging phase of global pipelines. We achieve state-of-the-art accuracy on four large-scale datasets.
  arXiv  Detail & Related papers  (2024-10-16T17:37:43Z)
• Vision Transformer with Sparse Scan Prior [24.78780746169092]
  We propose a textbfSparse textbfScan textbfSelf-textbfAttention mechanism ($rmS3rmA$)<n>This mechanism predefines a series of Anchors of Interest for each token and employs local attention to efficiently model the spatial information around these anchors.<n>Building on $rmS3rmA$, we introduce the textbfSparse textbfScan textbfVision
  arXiv  Detail & Related papers  (2024-05-22T04:34:36Z)
• Pyramid R-CNN: Towards Better Performance and Adaptability for 3D Object Detection [89.66162518035144]
  We present a flexible and high-performance framework, named Pyramid R-CNN, for two-stage 3D object detection from point clouds. We propose a novel second-stage module, named pyramid RoI head, to adaptively learn the features from the sparse points of interest. Our pyramid RoI head is robust to the sparse and imbalanced circumstances, and can be applied upon various 3D backbones to consistently boost the detection performance.
  arXiv  Detail & Related papers  (2021-09-06T14:17:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.