RayRoPE: Projective Ray Positional Encoding for Multi-view Attention

• URL: http://arxiv.org/abs/2601.15275v1
• Date: Wed, 21 Jan 2026 18:55:51 GMT
• Title: RayRoPE: Projective Ray Positional Encoding for Multi-view Attention
• Abstract summary: We study positional encodings for multi-view transformers that process tokens from a set of posed input images.<n>We seek a mechanism that encodes patches uniquely, allows SE(3)-invariant attention with multi-frequency similarity, and can be adaptive to the geometry of the underlying scene.<n>RayRoPE represents patch positions based on associated rays but leverages a predicted point along the ray instead of the direction for a geometry-aware encoding.
• Score: 46.94549341989407
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: We study positional encodings for multi-view transformers that process tokens from a set of posed input images, and seek a mechanism that encodes patches uniquely, allows SE(3)-invariant attention with multi-frequency similarity, and can be adaptive to the geometry of the underlying scene. We find that prior (absolute or relative) encoding schemes for multi-view attention do not meet the above desiderata, and present RayRoPE to address this gap. RayRoPE represents patch positions based on associated rays but leverages a predicted point along the ray instead of the direction for a geometry-aware encoding. To achieve SE(3) invariance, RayRoPE computes query-frame projective coordinates for computing multi-frequency similarity. Lastly, as the 'predicted' 3D point along a ray may not be precise, RayRoPE presents a mechanism to analytically compute the expected position encoding under uncertainty. We validate RayRoPE on the tasks of novel-view synthesis and stereo depth estimation and show that it consistently improves over alternate position encoding schemes (e.g. 15% relative improvement on LPIPS in CO3D). We also show that RayRoPE can seamlessly incorporate RGB-D input, resulting in even larger gains over alternatives that cannot positionally encode this information.

Related papers

• G-ray: Ray-Level Relative Geometric Position Encoding in Multi-View Vision Transformers under Camera Heterogeneity [44.82262218739513]
  We introduce G-ray, a ray-level relative position encoding whose rotary phases are parameterized by camera-local ray angles.<n>G-ray can be used directly or integrated with existing encodings, retaining complementary geometric cues without additional learned parameters.<n>We validate G-ray in three host encodings, RoPE, GTA, and RayRoPE, across 3D reconstruction and novel-view synthesis (NVS)
  arXiv  Detail & Related papers  (2026-09-14T04:30:34Z)
• RayViT: Ray-Conditioned Visual Representations for Viewpoint-Robust Imitation Learning [59.07474327224552]
  We propose bf RobotextRay-conditioned Vision Transformer (RayViT), a lightweight architecture that injects camera geometry into pretrained ViT backbones.<n>RayViT represents camera geometry as a Plcker ray map, patchifies it into ray features, and uses gated cross-attention to produce a ray-conditioned class token.<n> Experiments on sim- and real-robot tasks demonstrate that RayViT improves robustness by approximately 13 percentage points under camera perturbations.
  arXiv  Detail & Related papers  (2026-07-31T16:56:33Z)
• RayPE: Ray-Space Positional Encoding for 3D-Aware Video Generation [43.253222907352146]
  RayPE is a positional-encoding extension that injects per-token 6D Plucker coordinates additively into the queries and keys of self-attention.<n>The resulting attention score decomposes into a content term, a geometry term, and two content and geometry cross-terms.
  arXiv  Detail & Related papers  (2026-06-25T17:51:02Z)
• PEPS: Positional Encoding Projected Sampling -- Extended [1.096360026325105]
  Implicit neural representations (INRs) are increasingly being used as tools to map coordinates to signals.<n>Most INR methods are based on using high-dimensional projections of the initial coordinates through encoders such as grid or positional encoding.<n>We show that positional encoding can be used not only as high-dimensional embedding but as a series of meaningful points.
  arXiv  Detail & Related papers  (2026-04-27T08:23:39Z)
• URoPE: Universal Relative Position Embedding across Geometric Spaces [55.651792747815854]
  URoPE is a universal extension of Rotary Position Embedding (RoPE) to cross-view or cross-dimensional geometric spaces.<n>For each key/value image patch, URoPE samples 3D points along the corresponding camera ray at predefined depth anchors.<n>Standard 2D RoPE can then be applied using the projected pixel coordinates.
  arXiv  Detail & Related papers  (2026-04-20T18:52:03Z)
• Leveraging Transformer Decoder for Automotive Radar Object Detection [9.764772760421792]
  We present a Transformer-based architecture for 3D radar object detection that uses a novel Transformer Decoder.<n> Pyramid Token Fusion (PTF) converts a feature pyramid into a unified, scale-aware token sequence.<n>We evaluate the proposed framework on the RADDet, where it achieves significant improvements over state-of-the-art radar-only baselines.
  arXiv  Detail & Related papers  (2026-01-19T20:44:24Z)
• Cameras as Relative Positional Encoding [37.675563572777136]
  Multi-view transformers must use camera geometry to ground visual tokens in 3D space.<n>We show how relative camera conditioning improves performance in feedforward novel view synthesis.<n>We then verify that these benefits persist for different tasks, stereo depth estimation and discriminative cognition, as well as larger model sizes.
  arXiv  Detail & Related papers  (2025-07-14T17:22:45Z)
• PointVDP: Learning View-Dependent Projection by Fireworks Rays for 3D Point Cloud Segmentation [66.00721801098574]
  We propose view-dependent projection (VDP) to facilitate point cloud segmentation.<n>VDP generates data-driven projections from 3D point distributions.<n>We construct color regularization to optimize the framework.
  arXiv  Detail & Related papers  (2025-07-09T07:44:00Z)
• GTA: A Geometry-Aware Attention Mechanism for Multi-View Transformers [63.41460219156508]
  We argue that existing positional encoding schemes are suboptimal for 3D vision tasks. We propose a geometry-aware attention mechanism that encodes the geometric structure of tokens as relative transformation. We show that our attention, called Geometric Transform Attention (GTA), improves learning efficiency and performance of state-of-the-art transformer-based NVS models.
  arXiv  Detail & Related papers  (2023-10-16T13:16:09Z)
• Ray-Patch: An Efficient Querying for Light Field Transformers [10.859910783551937]
  We propose the Ray-Patch querying, a novel model to efficiently query transformers to decode implicit representations into target views. Our Ray-Patch decoding reduces the computational footprint and increases inference speed up to one order of magnitude compared to previous models.
  arXiv  Detail & Related papers  (2023-05-16T16:03:27Z)
• 3DPPE: 3D Point Positional Encoding for Multi-Camera 3D Object Detection Transformers [35.14784758217257]
  We introduce 3D point positional encoding, 3DPPE, to the 3D detection Transformer decoder. Despite the approximation, 3DPPE achieves 46.0 mAP and 51.4 NDS on the competitive nuScenes dataset.
  arXiv  Detail & Related papers  (2022-11-27T03:36:32Z)
• Inferring 3D change detection from bitemporal optical images [6.050310428775564]
  We propose two novel networks, able to solve simultaneously the 2D and 3D CD tasks. The aim of this work is to lay the foundations for the development of DL algorithms able to automatically infer an elevation (3D) CD map. The code and the 3DCD dataset are available at urlhttps://sites.google.com/uniroma1.it/3dchangedetection/home-page.
  arXiv  Detail & Related papers  (2022-05-31T15:53:33Z)
• Progressive Coordinate Transforms for Monocular 3D Object Detection [52.00071336733109]
  We propose a novel and lightweight approach, dubbed em Progressive Coordinate Transforms (PCT) to facilitate learning coordinate representations. In this paper, we propose a novel and lightweight approach, dubbed em Progressive Coordinate Transforms (PCT) to facilitate learning coordinate representations.
  arXiv  Detail & Related papers  (2021-08-12T15:22:33Z)
• Rethinking and Improving Relative Position Encoding for Vision Transformer [61.559777439200744]
  Relative position encoding (RPE) is important for transformer to capture sequence ordering of input tokens. We propose new relative position encoding methods dedicated to 2D images, called image RPE (iRPE)
  arXiv  Detail & Related papers  (2021-07-29T17:55:10Z)
• Learnable Fourier Features for Multi-DimensionalSpatial Positional Encoding [96.9752763607738]
  We propose a novel positional encoding method based on learnable Fourier features. Our experiments show that our learnable feature representation for multi-dimensional positional encoding outperforms existing methods.
  arXiv  Detail & Related papers  (2021-06-05T04:40:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.