Learning to Rotate: Temporal and Semantic Rotary Encoding for Sequential Modeling

• URL: http://arxiv.org/abs/2604.24717v1
• Date: Mon, 27 Apr 2026 17:26:15 GMT
• Title: Learning to Rotate: Temporal and Semantic Rotary Encoding for Sequential Modeling
• Abstract summary: We introduce SIREN-RoPE, which populates the rotation dimension with heterogeneous signals.<n>We evaluate on a production-scale news feed dataset using a generative recommender as the ranking model.
• Score: 6.277197086409669
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Every Transformer architecture dedicates enormous capacity to learning rich representations in semantic embedding space -- yet the rotation manifold acted upon by Rotary Positional Embeddings (RoPE) has been treated as a fixed, hand-crafted structure, populated only by discrete ordinal indices. We argue that this rotation space is a largely overlooked second dimension of expressivity in the attention mechanism, one whose systematic exploration may open a new door for attention-based architectures. The analogy to complex numbers is instructive: just as introducing the imaginary axis -- orthogonal to and independent of the real line -- unlocked new algebraic structure once believed impossible, treating the rotation manifold as a learnable, signal-conditioned space opens an orthogonal degree of freedom in attention. In this framing, the token embedding encodes the semantic (real) component of a representation -- what a token means -- while the rotation encodes its dynamic (imaginary) component -- how it relates to every other token across time, position, and context. We introduce SIREN-RoPE, a concrete instantiation of this idea, which populates the rotation dimension with heterogeneous signals -- continuous timestamps, cyclical temporal patterns, and categorical metadata -- via a dual-branch Sinusoidal Representation Network (SIREN). As a proof of concept, we evaluate on a production-scale news feed dataset from a major social network using a generative recommender as the ranking model, demonstrating that activating this hidden dimension yields consistent improvements across calibration and ranking objectives with negligible computational overhead. We invite the community to view the rotation space not as a solved positional-encoding detail, but as an untapped axis whose rich structure may prove as consequential for attention as the imaginary unit proved for algebra.

Related papers

• Enhancing Rotation-Invariant 3D Learning with Global Pose Awareness and Attention Mechanisms [30.575822636142956]
  We introduce the Shadow-informed Pose Feature (SiPF), which augments local RI descriptors with a globally consistent reference point (referred to as the'shadow') derived from a learned shared rotation.<n>We also propose Rotation-invariant Attention Convolution (RIAttnConv), an attention-based operator that integrates SiPFs into the feature aggregation process.
  arXiv  Detail & Related papers  (2025-11-11T23:01:28Z)
• Rotation Equivariant Arbitrary-scale Image Super-Resolution [62.41329042683779]
  The arbitrary-scale image super-resolution (ASISR) aims to achieve arbitrary-scale high-resolution recoveries from a low-resolution input image.<n>We make efforts to construct a rotation equivariant ASISR method in this study.
  arXiv  Detail & Related papers  (2025-08-07T08:51:03Z)
• Neuron: Learning Context-Aware Evolving Representations for Zero-Shot Skeleton Action Recognition [64.56321246196859]
  We propose a novel dyNamically Evolving dUal skeleton-semantic syneRgistic framework.<n>We first construct the spatial-temporal evolving micro-prototypes and integrate dynamic context-aware side information.<n>We introduce the spatial compression and temporal memory mechanisms to guide the growth of spatial-temporal micro-prototypes.
  arXiv  Detail & Related papers  (2024-11-18T05:16:11Z)
• PARE-Net: Position-Aware Rotation-Equivariant Networks for Robust Point Cloud Registration [8.668461141536383]
  Learning rotation-invariant distinctive features is a fundamental requirement for point cloud registration. Existing methods often use rotation-sensitive networks to extract features, while employing rotation augmentation to learn an approximate invariant mapping rudely. We propose a novel position-aware rotation-equivariant network, for efficient, light-weighted, and robust registration.
  arXiv  Detail & Related papers  (2024-07-14T10:26:38Z)
• Mesh Denoising Transformer [104.5404564075393]
  Mesh denoising is aimed at removing noise from input meshes while preserving their feature structures. SurfaceFormer is a pioneering Transformer-based mesh denoising framework. New representation known as Local Surface Descriptor captures local geometric intricacies. Denoising Transformer module receives the multimodal information and achieves efficient global feature aggregation.
  arXiv  Detail & Related papers  (2024-05-10T15:27:43Z)
• Robust Collaborative Perception without External Localization and Clock Devices [52.32342059286222]
  A consistent spatial-temporal coordination across multiple agents is fundamental for collaborative perception. Traditional methods depend on external devices to provide localization and clock signals. We propose a novel approach: aligning by recognizing the inherent geometric patterns within the perceptual data of various agents.
  arXiv  Detail & Related papers  (2024-05-05T15:20:36Z)
• Binding Dynamics in Rotating Features [72.80071820194273]
  We propose an alternative "cosine binding" mechanism, which explicitly computes the alignment between features and adjusts weights accordingly. This allows us to draw direct connections to self-attention and biological neural processes, and to shed light on the fundamental dynamics for object-centric representations to emerge in Rotating Features.
  arXiv  Detail & Related papers  (2024-02-08T12:31:08Z)
• Rotation-Invariant Transformer for Point Cloud Matching [42.5714375149213]
  We introduce RoITr, a Rotation-Invariant Transformer to cope with the pose variations in the point cloud matching task. We propose a global transformer with rotation-invariant cross-frame spatial awareness learned by the self-attention mechanism. RoITr surpasses the existing methods by at least 13 and 5 percentage points in terms of Inlier Ratio and Registration Recall.
  arXiv  Detail & Related papers  (2023-03-14T20:55:27Z)
• Solving Reasoning Tasks with a Slot Transformer [7.966351917016229]
  We present the Slot Transformer, an architecture that leverages slot attention, transformers and iterative variational inference on video scene data to infer representations. We evaluate the effectiveness of key components of the architecture, the model's representational capacity and its ability to predict from incomplete input.
  arXiv  Detail & Related papers  (2022-10-20T16:40:30Z)
• Attentive Rotation Invariant Convolution for Point Cloud-based Large Scale Place Recognition [11.433270318356675]
  We propose an Attentive Rotation Invariant Convolution (ARIConv) in this paper. We experimentally demonstrate that our model can achieve state-of-the-art performance on large scale place recognition task when the point cloud scans are rotated.
  arXiv  Detail & Related papers  (2021-08-29T09:10:56Z)
• Deep Parametric Continuous Convolutional Neural Networks [92.87547731907176]
  Parametric Continuous Convolution is a new learnable operator that operates over non-grid structured data. Our experiments show significant improvement over the state-of-the-art in point cloud segmentation of indoor and outdoor scenes.
  arXiv  Detail & Related papers  (2021-01-17T18:28:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.