Robo3R: Enhancing Robotic Manipulation with Accurate Feed-Forward 3D Reconstruction

• URL: http://arxiv.org/abs/2602.10101v1
• Date: Tue, 10 Feb 2026 18:58:15 GMT
• Title: Robo3R: Enhancing Robotic Manipulation with Accurate Feed-Forward 3D Reconstruction
• Abstract summary: 3D spatial perception is fundamental to general robotic manipulation, yet obtaining reliable, high-quality 3D geometry remains challenging.<n>We introduce Robo3R, a manipulation-ready 3D reconstruction model that predicts accurate, metric-scale scene geometry directly from RGB images and robot states in real time.<n>We observe consistent gains in performance, suggesting the promise of this alternative 3D sensing module for robotic manipulation.
• Score: 70.06600045165905
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: 3D spatial perception is fundamental to generalizable robotic manipulation, yet obtaining reliable, high-quality 3D geometry remains challenging. Depth sensors suffer from noise and material sensitivity, while existing reconstruction models lack the precision and metric consistency required for physical interaction. We introduce Robo3R, a feed-forward, manipulation-ready 3D reconstruction model that predicts accurate, metric-scale scene geometry directly from RGB images and robot states in real time. Robo3R jointly infers scale-invariant local geometry and relative camera poses, which are unified into the scene representation in the canonical robot frame via a learned global similarity transformation. To meet the precision demands of manipulation, Robo3R employs a masked point head for sharp, fine-grained point clouds, and a keypoint-based Perspective-n-Point (PnP) formulation to refine camera extrinsics and global alignment. Trained on Robo3R-4M, a curated large-scale synthetic dataset with four million high-fidelity annotated frames, Robo3R consistently outperforms state-of-the-art reconstruction methods and depth sensors. Across downstream tasks including imitation learning, sim-to-real transfer, grasp synthesis, and collision-free motion planning, we observe consistent gains in performance, suggesting the promise of this alternative 3D sensing module for robotic manipulation.

Related papers

• MeshMimic: Geometry-Aware Humanoid Motion Learning through 3D Scene Reconstruction [54.36564144414704]
  MeshMimic is an innovative framework that bridges 3D scene reconstruction and embodied intelligence to enable humanoid robots to learn coupled "motion-terrain" interactions directly from video.<n>By leveraging state-of-the-art 3D vision models, our framework precisely segments and reconstructs both human trajectories and the underlying 3D geometry of terrains and objects.
  arXiv  Detail & Related papers  (2026-02-17T17:09:45Z)
• Fiducial Exoskeletons: Image-Centric Robot State Estimation [21.491677821308688]
  We introduce Fiducial Exoskeletons, an image-based reformulation of 3D robot state estimation.<n>Our key insight is twofold. First, we cast robot state estimation as 6D pose estimation of each link from a single RGB image.<n>Second, we make per-link 6D pose estimation robust and simple - even without learning.
  arXiv  Detail & Related papers  (2026-01-12T22:04:25Z)
• A High-Fidelity Digital Twin for Robotic Manipulation Based on 3D Gaussian Splatting [0.0]
  We present a framework that constructs high-quality digital twins within minutes from sparse RGB inputs.<n>Our system employs 3D Gaussian Splatting (3DGS) for fast, photorealistic reconstruction as a unified scene representation.<n>In experiments with a Franka Emika Panda robot, we demonstrate that this enhanced geometric accuracy effectively supports robust manipulation in real-world trials.
  arXiv  Detail & Related papers  (2026-01-06T17:29:10Z)
• RePose-NeRF: Robust Radiance Fields for Mesh Reconstruction under Noisy Camera Poses [0.5328562411561214]
  We propose a robust framework that reconstructs high-quality, editable 3D meshes directly from multi-view images.<n>Our approach jointly refines camera poses while learning an implicit scene representation that captures fine geometric detail and appearance.
  arXiv  Detail & Related papers  (2025-11-11T18:25:58Z)
• HOSt3R: Keypoint-free Hand-Object 3D Reconstruction from RGB images [27.025336665386735]
  We propose a robust, keypoint detector-free approach to estimating hand-object 3D transformations from monocular motion video/images.<n>We further integrate this with a multi-view reconstruction pipeline to accurately recover hand-object 3D shape.<n>Our method, named HOSt3R, is unconstrained, does not rely on pre-scanned object templates or camera intrinsics, and reaches state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-08-22T15:30:40Z)
• HORT: Monocular Hand-held Objects Reconstruction with Transformers [61.36376511119355]
  Reconstructing hand-held objects in 3D from monocular images is a significant challenge in computer vision.<n>We propose a transformer-based model to efficiently reconstruct dense 3D point clouds of hand-held objects.<n>Our method achieves state-of-the-art accuracy with much faster inference speed, while generalizing well to in-the-wild images.
  arXiv  Detail & Related papers  (2025-03-27T09:45:09Z)
• Stereo4D: Learning How Things Move in 3D from Internet Stereo Videos [76.07894127235058]
  We present a system for mining high-quality 4D reconstructions from internet stereoscopic, wide-angle videos.<n>We use this method to generate large-scale data in the form of world-consistent, pseudo-metric 3D point clouds.<n>We demonstrate the utility of this data by training a variant of DUSt3R to predict structure and 3D motion from real-world image pairs.
  arXiv  Detail & Related papers  (2024-12-12T18:59:54Z)
• SUGAR: Pre-training 3D Visual Representations for Robotics [85.55534363501131]
  We introduce a novel 3D pre-training framework for robotics named SUGAR. SUGAR captures semantic, geometric and affordance properties of objects through 3D point clouds. We show that SUGAR's 3D representation outperforms state-of-the-art 2D and 3D representations.
  arXiv  Detail & Related papers  (2024-04-01T21:23:03Z)
• Neural Scene Representation for Locomotion on Structured Terrain [56.48607865960868]
  We propose a learning-based method to reconstruct the local terrain for a mobile robot traversing urban environments. Using a stream of depth measurements from the onboard cameras and the robot's trajectory, the estimates the topography in the robot's vicinity. We propose a 3D reconstruction model that faithfully reconstructs the scene, despite the noisy measurements and large amounts of missing data coming from the blind spots of the camera arrangement.
  arXiv  Detail & Related papers  (2022-06-16T10:45:17Z)
• Amodal 3D Reconstruction for Robotic Manipulation via Stability and Connectivity [3.359622001455893]
  Learning-based 3D object reconstruction enables single- or few-shot estimation of 3D object models. Existing 3D reconstruction techniques optimize for visual reconstruction fidelity, typically measured by chamfer distance or voxel IOU. We propose ARM, an amodal 3D reconstruction system that introduces a stability prior over object shapes, (2) a connectivity prior, and (3) a multi-channel input representation.
  arXiv  Detail & Related papers  (2020-09-28T08:52:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.