Higher-Dimensional Rotary Position Embedding

• URL: http://arxiv.org/abs/2608.29715v1
• Date: Sun, 30 Aug 2026 10:46:24 GMT
• Title: Higher-Dimensional Rotary Position Embedding
• Abstract summary: Rotary Position Embedding (RoPE) embeds positional information with independent 2D rotations, forming relative position terms in self-attention.<n>We propose HD-RoPE, which extends RoPE from independent 2D rotations to higher-dimensional rotations.<n> HD-RoPE is easily optimized for engineering efficiency without introducing additional trainable parameters.
• Score: 36.55400843027212
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Transformers rely on position embedding mechanisms in long context modeling in most cases. Rotary Position Embedding (RoPE) embeds positional information with independent 2D rotations, forming relative position terms in self-attention. However, its pairwise, block-based, and decoupled structure limits deep mixing and robustness across channels. We propose HD-RoPE, which extends RoPE from independent 2D rotations to higher-dimensional rotations and introduces a Paley-I orthogonal basis to obtain balanced, isotropic, and dense phase mixing within each rotation subspace. This significantly enhances channel coupling and rotational degrees of freedom while maintaining orthogonal stability and the relative position closure property. Furthermore, HD-RoPE is easily optimized for engineering efficiency without introducing additional trainable parameters. We have conducted extensive evaluation results demonstrating that HD-RoPE achieves significant performance improvements over standard RoPE across various popular benchmarks and in both long and short contexts.

Related papers

• Disentangling the Expressivity of RoPE [53.85490969529292]
  We formalize accounts for fully uniform, finite-precision soft-attention transformers.<n>We find that, if every rotary component is periodic, RoPE transformers recognize exactly the languages definable in past temporal logic with modular predicates.
  arXiv  Detail & Related papers  (2026-08-12T10:37:21Z)
• nD-RoPE: A Generalized RoPE for n-Dimensional Position Embedding [7.04717162882032]
  nD-RoPE is a decomposition-free generalization of RoPE to arbitrary dimensions.<n> Experiments across images, videos, and point clouds demonstrate consistent performance gains and improved generalization in high-dimensional settings.
  arXiv  Detail & Related papers  (2026-06-10T14:38:00Z)
• RoVE: Rotary Value Embeddings Attention for Relative Position-dependent Value Pathways [52.94485753759119]
  Rotary Position Embeddings (RoPE) make attention scores position-relative but leave the value pathway position-blind.<n>We propose RoVE, a parameter-free modification that makes values position-sensitive by rotating them simultaneously with keys, and show that it turns RoPE attention into attentive convolution.
  arXiv  Detail & Related papers  (2026-06-09T09:56:50Z)
• Spiral RoPE: Rotate Your Rotary Positional Embeddings in the 2D Plane [49.14270539697387]
  Spiral RoPE is a simple yet effective extension that enables multi-directional positional encoding.<n>Across a wide range of vision tasks including classification, segmentation, and generation, Spiral RoPE consistently improves performance.
  arXiv  Detail & Related papers  (2026-02-03T07:56:58Z)
• One Attention, One Scale: Phase-Aligned Rotary Positional Embeddings for Mixed-Resolution Diffusion Transformer [48.30024190686566]
  Cross-Resolution Phase-Aligned Attention (CRPA) is a training-free drop-in fix that eliminates this failure at its source.<n>CRPA is fully compatible with pretrained DiTs, stabilizes all heads and layers uniformly.<n>We demonstrate that CRPA enables high-fidelity and efficient mixed-resolution generation, outperforming previous state-of-the-art methods on image and video generation.
  arXiv  Detail & Related papers  (2025-11-24T23:10:15Z)
• Selective Rotary Position Embedding [84.22998043041198]
  We introduce textitSelective RoPE, an textitinput-dependent rotary embedding mechanism.<n>We show that softmax attention already performs a hidden form of these rotations on query-key pairs.<n>We validate our method by equipping gated transformers with textitSelective RoPE, demonstrating that its input-dependent rotations improve performance in language modeling.
  arXiv  Detail & Related papers  (2025-11-21T16:50:00Z)
• Positional Encoding via Token-Aware Phase Attention [45.855203550592734]
  We show that Rotary Positional Embedding (RoPE) introduces an intrinsic distance-dependent bias in attention scores that limits RoPE's ability to model long-context.<n>This paper introduces Token-Aware Phase Attention (TAPA), a new positional encoding method that incorporates a learnable phase function into the attention mechanism.
  arXiv  Detail & Related papers  (2025-09-16T03:53:32Z)
• ComRoPE: Scalable and Robust Rotary Position Embedding Parameterized by Trainable Commuting Angle Matrices [25.99231204405503]
  We propose ComRoPE, which generalizes Rotary Positional PE (RoPE) by defining it in terms of trainable commuting angle matrices.<n>We present two types of trainable commuting angle matrices as sufficient solutions to the RoPE equation.<n>Our framework shows versatility in generalizing to existing RoPE formulations and offering new insights for future positional encoding research.
  arXiv  Detail & Related papers  (2025-06-04T09:10:02Z)
• Rethinking RoPE: A Mathematical Blueprint for N-dimensional Positional Embedding [1.8142288667655782]
  We propose a systematic mathematical framework for Rotary Position Embedding (RoPE)<n>We derive the necessary and sufficient conditions for any valid $N$-dimensional RoPE based on two core properties of RoPE - relativity and reversibility.<n>Our framework unifies and explains existing RoPE designs while enabling principled extensions to higher-dimensional modalities and tasks.
  arXiv  Detail & Related papers  (2025-04-07T21:58:22Z)
• DRoPE: Directional Rotary Position Embedding for Efficient Agent Interaction Modeling [9.86959003425198]
  Directional Rotary Position Embedding (DRoPE) is a novel adaptation of Rotary Position Embedding (RoPE) originally developed in natural language processing.<n>DRoPE overcomes limitations by introducing a uniform identity scalar into RoPE's 2D rotary transformation.<n> Empirical evaluations confirm DRoPE's good performance and significantly reduced space complexity.
  arXiv  Detail & Related papers  (2025-03-19T09:23:09Z)
• Rotation-Invariant Transformer for Point Cloud Matching [42.5714375149213]
  We introduce RoITr, a Rotation-Invariant Transformer to cope with the pose variations in the point cloud matching task. We propose a global transformer with rotation-invariant cross-frame spatial awareness learned by the self-attention mechanism. RoITr surpasses the existing methods by at least 13 and 5 percentage points in terms of Inlier Ratio and Registration Recall.
  arXiv  Detail & Related papers  (2023-03-14T20:55:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.