RayViT: Ray-Conditioned Visual Representations for Viewpoint-Robust Imitation Learning

• URL: http://arxiv.org/abs/2607.29622v1
• Date: Fri, 31 Jul 2026 16:56:33 GMT
• Title: RayViT: Ray-Conditioned Visual Representations for Viewpoint-Robust Imitation Learning
• Abstract summary: We propose bf RobotextRay-conditioned Vision Transformer (RayViT), a lightweight architecture that injects camera geometry into pretrained ViT backbones.<n>RayViT represents camera geometry as a Plcker ray map, patchifies it into ray features, and uses gated cross-attention to produce a ray-conditioned class token.<n> Experiments on sim- and real-robot tasks demonstrate that RayViT improves robustness by approximately 13 percentage points under camera perturbations.
• Score: 59.07474327224552
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Visual imitation learning enables robots to acquire visuomotor skills directly from images, yet RGB observations lack explicit geometric cues, making learned policies brittle to camera perturbations. To address this, we propose \textbf{Ray-conditioned Vision Transformer Encoder (RayViT)}, a lightweight architecture that injects camera geometry into pretrained ViT backbones. RayViT represents camera geometry as a Plücker ray map, patchifies it into ray features, and uses gated cross-attention to produce a ray-conditioned class token. These ray features are added as dense positional embeddings, while the ray class token replaces the original ViT class token to provide a geometry-aware summary representation. We combine this approach with an auxiliary cosine similarity loss to consistently improve the performance and robustness for geometry-aware tokens. Experiments on sim- and real-robot tasks demonstrate that RayViT improves robustness by approximately 13 percentage points under camera perturbations in multi-task RoboCasa benchmark and by 1.78 average completed stages in real-world multi-task success rate compared to baselines.

Related papers

• RayLift: Lifting Complementary Ray-Wise Evidence with 3D Geometry Priors for Semantic Scene Completion [23.7438085195588]
  Camera-based 3D semantic scene completion (SSC) provides comprehensive scene understanding for autonomous driving and robotics.<n>Existing methods often treat stereo depth estimates as deterministic geometric constraints, causing depth uncertainty and local correspondence errors to propagate directly into voxel representations.<n>We propose RayLift, a framework that uses stereo geometry as a metric reference while incorporating complementary ray evidence to recover reliable 3D structures adaptively.
  arXiv  Detail & Related papers  (2026-08-09T04:41:34Z)
• G$^3$VLA: Geometric inductive bias for Vision-Language-Action Models [3.704517635293094]
  Vision-language-action (VLA) models have made rapid progress in generalist robot manipulation.<n>Their visual tokens remain grounded in 2D image coordinates rather than the calibrated geometry of the robot's cameras.<n>We propose G$3$VLA, a camera-aware geometric module that injects calibrated structure into the visual-token stream of a pretrained VLA.
  arXiv  Detail & Related papers  (2026-06-23T12:02:36Z)
• From Rays to Projections: Better Inputs for Feed-Forward View Synthesis [26.130973137744018]
  Feed-forward view synthesis models predict a novel view in a single pass with minimal 3D inductive bias.<n>Existing works encode cameras as Plcker ray maps, which tie predictions to the arbitrary world coordinate gauge and make them sensitive to small camera transformations.<n>We propose projective conditioning, which replaces raw camera parameters with a target-view projective cue that provides a stable 2D input.
  arXiv  Detail & Related papers  (2026-01-08T17:03:44Z)
• E-RayZer: Self-supervised 3D Reconstruction as Spatial Visual Pre-training [55.61379509038588]
  We present E-RayZer, a self-supervised large 3D Vision model that learns truly 3D-aware representations directly from unlabeled images.<n>E-RayZer operates directly in 3D space, performing self-supervised 3D reconstruction with Explicit geometry.
  arXiv  Detail & Related papers  (2025-12-11T18:59:53Z)
• RaySt3R: Predicting Novel Depth Maps for Zero-Shot Object Completion [49.933001840775816]
  RaySt3R recasts 3D shape completion as a novel view synthesis problem.<n>We train a feedforward transformer to predict depth maps, object masks, and per-pixel confidence scores for query rays.<n>RaySt3R fuses these predictions across multiple query views to reconstruct complete 3D shapes.
  arXiv  Detail & Related papers  (2025-06-05T17:43:23Z)
• LaRI: Layered Ray Intersections for Single-view 3D Geometric Reasoning [75.9814389360821]
  layered ray intersections (LaRI) is a new method for unseen geometry reasoning from a single image.<n>Benefiting from the compact and layered representation, LaRI enables complete, efficient, and view-aligned geometric reasoning.<n>We build a complete training data generation pipeline for synthetic and real-world data, including 3D objects and scenes.
  arXiv  Detail & Related papers  (2025-04-25T15:31:29Z)
• RangeViT: Towards Vision Transformers for 3D Semantic Segmentation in Autonomous Driving [80.14669385741202]
  Vision transformers (ViTs) have achieved state-of-the-art results in many image-based benchmarks. ViTs are notoriously hard to train and require a lot of training data to learn powerful representations. We show that our method, called RangeViT, outperforms existing projection-based methods on nuScenes and Semantic KITTI.
  arXiv  Detail & Related papers  (2023-01-24T18:50:48Z)
• Ray-ONet: Efficient 3D Reconstruction From A Single RGB Image [16.21925131245367]
  We propose Ray-ONet to reconstruct detailed 3D models from monocular images efficiently. By predicting a series of occupancy probabilities along a ray that is back-projected from a pixel in the camera coordinate, our method Ray-ONet improves the reconstruction accuracy in comparison with Occupancy Networks (ONet)
  arXiv  Detail & Related papers  (2021-07-05T09:45:57Z)
• Lightweight Multi-View 3D Pose Estimation through Camera-Disentangled Representation [57.11299763566534]
  We present a solution to recover 3D pose from multi-view images captured with spatially calibrated cameras. We exploit 3D geometry to fuse input images into a unified latent representation of pose, which is disentangled from camera view-points. Our architecture then conditions the learned representation on camera projection operators to produce accurate per-view 2d detections.
  arXiv  Detail & Related papers  (2020-04-05T12:52:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.