nD-RoPE: A Generalized RoPE for n-Dimensional Position Embedding

• URL: http://arxiv.org/abs/2606.12146v1
• Date: Wed, 10 Jun 2026 14:38:00 GMT
• Title: nD-RoPE: A Generalized RoPE for n-Dimensional Position Embedding
• Abstract summary: nD-RoPE is a decomposition-free generalization of RoPE to arbitrary dimensions.<n> Experiments across images, videos, and point clouds demonstrate consistent performance gains and improved generalization in high-dimensional settings.
• Score: 7.04717162882032
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Rotary Position Embedding (RoPE) is widely adopted in Transformer models, yet its extension to high-dimensional domains lacks a unified theoretical formulation. Most existing approaches either apply rotations independently along each axis or empirically mix frequencies, which limits cross-dimensional interactions and yields direction-dependent representations. To address these limitations, we propose nD-RoPE, a decomposition-free generalization of RoPE to arbitrary dimensions. From a translation-invariant formulation in continuous Hilbert space, we derive a spectral condition for isotropy that requires treating positions and frequencies as coupled \(n\)-dimensional vectors. We instantiate this formulation with a multi-scale regular-simplex wave-vector design, which provides non-degenerate spatial coverage and a symmetric, directionally balanced second-order response. Experiments across images, videos, and point clouds demonstrate consistent performance gains and improved generalization in high-dimensional settings.

Related papers

• Rotating-wave approximation for spin-boson models with structured fields [0.0]
  We derive state-dependent bounds on the difference between two quantum evolutions generated by Hamiltonians sharing a common form domain.<n>We prove a quantitative bound on the rotating-wave approximation for spin-boson models with a structured boson field.
  arXiv  Detail & Related papers  (2026-08-31T17:00:03Z)
• Disentangling the Expressivity of RoPE [53.85490969529292]
  We formalize accounts for fully uniform, finite-precision soft-attention transformers.<n>We find that, if every rotary component is periodic, RoPE transformers recognize exactly the languages definable in past temporal logic with modular predicates.
  arXiv  Detail & Related papers  (2026-08-12T10:37:21Z)
• General Covariant Action Modeling: Constructing Generalized Manifolds via Spatio-Temporal Decoupling [48.559681359374146]
  Generalized Action Manifold (GAM) framework enforces general covariance through structural disentanglement.<n>GAM enables superior transfer and robustness capabilities, outperforming geometry-agnostic baselines.
  arXiv  Detail & Related papers  (2026-05-27T03:38:15Z)
• Rethinking Diffusion Models with Symmetries through Canonicalization with Applications to Molecular Graph Generation [56.361076943802594]
  CanonFlow achieves state-of-the-art performance on the challenging GEOM-DRUG dataset, and the advantage remains large in few-step generation.
  arXiv  Detail & Related papers  (2026-02-16T18:58:55Z)
• Deterministic Generation of Arbitrary Fock States via Resonant Subspace Engineering [9.529916381983394]
  We introduce resonant subspace engineering (RSE), a protocol that analytically confines the infinite-dimensional bosonic dynamics to a two-dimensional invariant subspace.<n>RSE achieves $O(n1/4)$ scaling in both evolution time and gate depth, showing a fundamental improvement over existing deterministic schemes.<n>RSE provides a scalable and analytically transparent framework for large-scale bosonic state engineering and gate synthesis.
  arXiv  Detail & Related papers  (2026-02-12T16:36:58Z)
• Parallel Complex Diffusion for Scalable Time Series Generation [50.01609741902786]
  PaCoDi is a spectral-native architecture that decouples generative modeling in the frequency domain.<n>We show that PaCoDi outperforms existing baselines in both generation quality and inference speed.
  arXiv  Detail & Related papers  (2026-02-10T14:31:53Z)
• Spiral RoPE: Rotate Your Rotary Positional Embeddings in the 2D Plane [49.14270539697387]
  Spiral RoPE is a simple yet effective extension that enables multi-directional positional encoding.<n>Across a wide range of vision tasks including classification, segmentation, and generation, Spiral RoPE consistently improves performance.
  arXiv  Detail & Related papers  (2026-02-03T07:56:58Z)
• ARGUS: Adaptive Rotation-Invariant Geometric Unsupervised System [0.0]
  This paper introduces Argus, a framework that reconceptualizes drift detection as tracking local statistics over a fixed spatial partition of the data manifold.<n> Voronoi tessellations over canonical orthonormal frames yield drift metrics that are invariant to transformations.<n>A graph-theoretic characterization of drift propagation is developed that distinguishes coherent distributional shifts from isolated perturbations.
  arXiv  Detail & Related papers  (2026-01-03T22:39:20Z)
• WUSH: Near-Optimal Adaptive Transforms for LLM Quantization [52.77441224845925]
  Quantization to low bitwidth is a standard approach for deploying large language models.<n>A few extreme weights and activations stretch the dynamic range and reduce the effective resolution of the quantizer.<n>We derive, for the first time, closed-form optimal linear blockwise transforms for joint weight-activation quantization.
  arXiv  Detail & Related papers  (2025-11-30T16:17:34Z)
• Rethinking RoPE: A Mathematical Blueprint for N-dimensional Positional Embedding [1.8142288667655782]
  We propose a systematic mathematical framework for Rotary Position Embedding (RoPE)<n>We derive the necessary and sufficient conditions for any valid $N$-dimensional RoPE based on two core properties of RoPE - relativity and reversibility.<n>Our framework unifies and explains existing RoPE designs while enabling principled extensions to higher-dimensional modalities and tasks.
  arXiv  Detail & Related papers  (2025-04-07T21:58:22Z)
• SGFormer: Spherical Geometry Transformer for 360 Depth Estimation [52.23806040289676]
  Panoramic distortion poses a significant challenge in 360 depth estimation.<n>We propose a spherical geometry transformer, named SGFormer, to address the above issues.<n>We also present a query-based global conditional position embedding to compensate for spatial structure at varying resolutions.
  arXiv  Detail & Related papers  (2024-04-23T12:36:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.