G-ray: Ray-Level Relative Geometric Position Encoding in Multi-View Vision Transformers under Camera Heterogeneity

• URL: http://arxiv.org/abs/2609.15018v2
• Date: Tue, 15 Sep 2026 11:44:01 GMT
• Title: G-ray: Ray-Level Relative Geometric Position Encoding in Multi-View Vision Transformers under Camera Heterogeneity
• Abstract summary: We introduce G-ray, a ray-level relative position encoding whose rotary phases are parameterized by camera-local ray angles.<n>G-ray can be used directly or integrated with existing encodings, retaining complementary geometric cues without additional learned parameters.<n>We validate G-ray in three host encodings, RoPE, GTA, and RayRoPE, across 3D reconstruction and novel-view synthesis (NVS)
• Score: 44.82262218739513
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We study relative position encoding for multi-view vision Transformers under camera heterogeneity, including varying fields of view (FoVs) or projection models. Existing rotary relative position encodings commonly use image-plane positional coordinates, producing projection-dependent relative phases and inconsistent geometric cues for cross-projection attention. We introduce G-ray, a ray-level relative position encoding whose rotary phases are parameterized by camera-local ray angles. The same camera-local ray pair induces the same relative phase across projections, providing projection-invariant positional consistency. G-ray can be used directly or integrated with existing encodings, retaining complementary geometric cues without additional learned parameters. We validate G-ray in three host encodings, RoPE, GTA, and RayRoPE, across 3D reconstruction and novel-view synthesis (NVS). Across three heterogeneous 3D reconstruction benchmarks at 50 views, G-ray leads all six averaged metrics and reduces mean pointmap relative error by 45.8% over MapAnything, with calibration supplied to both. Trained exclusively on homogeneous pinhole images, the 3D reconstruction model handles mixed pinhole and non-pinhole inputs without retraining and remains competitive on homogeneous pinhole 3D reconstruction protocols. For NVS, GTA and RayRoPE improve with G-ray under joint viewpoint and FoV variation. The project's webpage is available at https://g-ray-project.github.io/.

Related papers

• RayPE: Ray-Space Positional Encoding for 3D-Aware Video Generation [43.253222907352146]
  RayPE is a positional-encoding extension that injects per-token 6D Plucker coordinates additively into the queries and keys of self-attention.<n>The resulting attention score decomposes into a content term, a geometry term, and two content and geometry cross-terms.
  arXiv  Detail & Related papers  (2026-06-25T17:51:02Z)
• SCOPE: Scale-Consistent One-Pass Estimation of 3D Geometry [69.89196162649091]
  SCOPE (Scale-Consistent One-Pass Estimation of 3D Geometry) is a novel approach for estimating 3D geometry from extended monocular video sequences.<n>Our approach generates affine-invariant 3D point maps with shared parameters across entire sequences, enabling consistent scale-invariant representations.
  arXiv  Detail & Related papers  (2026-06-19T10:23:25Z)
• OmniDrive: An LLM-Choreographed Multi-Agent World Model with Unified Latent Co-Compression for Multi-View Driving Video Generation [23.42968075775045]
  DRIVE-CHOREO recasts controllable multi-view video generation as latent choreography.<n>On nuScenes, DRIVE-CHOREO sets new state-of-the-art multi-view consistency and BEV mAP (21.6) with competitive FVD (45.7); a detector trained purely on our synthetic data gains +2.4 NDS on the real validation split, validating downstream utility.
  arXiv  Detail & Related papers  (2026-06-16T05:25:55Z)
• IVGT: Implicit Visual Geometry Transformer for Neural Scene Representation [76.36174247570716]
  We propose IVGT, an Implicit Visual Geometry Transformer that implicitly models continuous and coherent geometry from pose-free multi-view images.<n> IVGT learns a continuous neural scene representation in a canonical coordinate system and supports continuous spatial queries at any 3D positions.<n>It allows direct extraction of continuous and coherent surface geometry, enabling rendering of RGB images, depth maps, and surface normal maps from arbitrary viewpoints.
  arXiv  Detail & Related papers  (2026-05-15T17:59:57Z)
• URoPE: Universal Relative Position Embedding across Geometric Spaces [55.651792747815854]
  URoPE is a universal extension of Rotary Position Embedding (RoPE) to cross-view or cross-dimensional geometric spaces.<n>For each key/value image patch, URoPE samples 3D points along the corresponding camera ray at predefined depth anchors.<n>Standard 2D RoPE can then be applied using the projected pixel coordinates.
  arXiv  Detail & Related papers  (2026-04-20T18:52:03Z)
• RayRoPE: Projective Ray Positional Encoding for Multi-view Attention [46.94549341989407]
  We study positional encodings for multi-view transformers that process tokens from a set of posed input images.<n>We seek a mechanism that encodes patches uniquely, allows SE(3)-invariant attention with multi-frequency similarity, and can be adaptive to the geometry of the underlying scene.<n>RayRoPE represents patch positions based on associated rays but leverages a predicted point along the ray instead of the direction for a geometry-aware encoding.
  arXiv  Detail & Related papers  (2026-01-21T18:55:51Z)
• LaRI: Layered Ray Intersections for Single-view 3D Geometric Reasoning [75.9814389360821]
  layered ray intersections (LaRI) is a new method for unseen geometry reasoning from a single image.<n>Benefiting from the compact and layered representation, LaRI enables complete, efficient, and view-aligned geometric reasoning.<n>We build a complete training data generation pipeline for synthetic and real-world data, including 3D objects and scenes.
  arXiv  Detail & Related papers  (2025-04-25T15:31:29Z)
• LucidFusion: Reconstructing 3D Gaussians with Arbitrary Unposed Images [23.96972213606037]
  We reformulate 3D reconstruction as image-to-image translation and introduce the Relative Coordinate Map (RCM)<n>RCM aligns multiple unposed images to a main view without pose estimation.<n>While RCM simplifies the process, its lack of global 3D supervision can yield noisy outputs.<n>Our LucidFusion framework handles an arbitrary number of unposed inputs, producing robust 3D reconstructions within seconds and paving the way for more flexible, pose-free 3D pipelines.
  arXiv  Detail & Related papers  (2024-10-21T04:47:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.