RaySt3R: Predicting Novel Depth Maps for Zero-Shot Object Completion

• URL: http://arxiv.org/abs/2506.05285v1
• Date: Thu, 05 Jun 2025 17:43:23 GMT
• Title: RaySt3R: Predicting Novel Depth Maps for Zero-Shot Object Completion
• Authors: Bardienus P. Duisterhof, Jan Oberst, Bowen Wen, Stan Birchfield, Deva Ramanan, Jeffrey Ichnowski,
• Abstract summary: RaySt3R recasts 3D shape completion as a novel view synthesis problem.<n>We train a feedforward transformer to predict depth maps, object masks, and per-pixel confidence scores for query rays.<n>RaySt3R fuses these predictions across multiple query views to reconstruct complete 3D shapes.
• Score: 49.933001840775816
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: 3D shape completion has broad applications in robotics, digital twin reconstruction, and extended reality (XR). Although recent advances in 3D object and scene completion have achieved impressive results, existing methods lack 3D consistency, are computationally expensive, and struggle to capture sharp object boundaries. Our work (RaySt3R) addresses these limitations by recasting 3D shape completion as a novel view synthesis problem. Specifically, given a single RGB-D image and a novel viewpoint (encoded as a collection of query rays), we train a feedforward transformer to predict depth maps, object masks, and per-pixel confidence scores for those query rays. RaySt3R fuses these predictions across multiple query views to reconstruct complete 3D shapes. We evaluate RaySt3R on synthetic and real-world datasets, and observe it achieves state-of-the-art performance, outperforming the baselines on all datasets by up to 44% in 3D chamfer distance. Project page: https://rayst3r.github.io

Related papers

• E3D-Bench: A Benchmark for End-to-End 3D Geometric Foundation Models [78.1674905950243]
  We present the first comprehensive benchmark for 3D geometric foundation models (GFMs)<n>GFMs directly predict dense 3D representations in a single feed-forward pass, eliminating the need for slow or unavailable precomputed camera parameters.<n>We evaluate 16 state-of-the-art GFMs, revealing their strengths and limitations across tasks and domains.<n>All code, evaluation scripts, and processed data will be publicly released to accelerate research in 3D spatial intelligence.
  arXiv  Detail & Related papers  (2025-06-02T17:53:09Z)
• LaRI: Layered Ray Intersections for Single-view 3D Geometric Reasoning [75.9814389360821]
  layered ray intersections (LaRI) is a new method for unseen geometry reasoning from a single image.<n>Benefiting from the compact and layered representation, LaRI enables complete, efficient, and view-aligned geometric reasoning.<n>We build a complete training data generation pipeline for synthetic and real-world data, including 3D objects and scenes.
  arXiv  Detail & Related papers  (2025-04-25T15:31:29Z)
• Zero-Shot Multi-Object Scene Completion [59.325611678171974]
  We present a 3D scene completion method that recovers the complete geometry of multiple unseen objects in complex scenes from a single RGB-D image. Our method outperforms the current state-of-the-art on both synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2024-03-21T17:59:59Z)
• RayDF: Neural Ray-surface Distance Fields with Multi-view Consistency [10.55497978011315]
  We propose a new framework called RayDF to formulate 3D shapes as ray-based neural functions. Our method achieves a 1000x faster speed than coordinate-based methods to render an 800x800 depth image.
  arXiv  Detail & Related papers  (2023-10-30T15:22:50Z)
• MobileBrick: Building LEGO for 3D Reconstruction on Mobile Devices [78.20154723650333]
  High-quality 3D ground-truth shapes are critical for 3D object reconstruction evaluation. We introduce a novel multi-view RGBD dataset captured using a mobile device. We obtain precise 3D ground-truth shape without relying on high-end 3D scanners.
  arXiv  Detail & Related papers  (2023-03-03T14:02:50Z)
• OmniObject3D: Large-Vocabulary 3D Object Dataset for Realistic Perception, Reconstruction and Generation [107.71752592196138]
  We propose OmniObject3D, a large vocabulary 3D object dataset with massive high-quality real-scanned 3D objects. It comprises 6,000 scanned objects in 190 daily categories, sharing common classes with popular 2D datasets. Each 3D object is captured with both 2D and 3D sensors, providing textured meshes, point clouds, multiview rendered images, and multiple real-captured videos.
  arXiv  Detail & Related papers  (2023-01-18T18:14:18Z)
• HM3D-ABO: A Photo-realistic Dataset for Object-centric Multi-view 3D Reconstruction [37.29140654256627]
  We present a photo-realistic object-centric dataset HM3D-ABO. It is constructed by composing realistic indoor scene and realistic object. The dataset could also be useful for tasks such as camera pose estimation and novel-view synthesis.
  arXiv  Detail & Related papers  (2022-06-24T16:02:01Z)
• CoReNet: Coherent 3D scene reconstruction from a single RGB image [43.74240268086773]
  We build on advances in deep learning to reconstruct the shape of a single object given only one RBG image as input. We propose three extensions: (1) ray-traced skip connections that propagate local 2D information to the output 3D volume in a physically correct manner; (2) a hybrid 3D volume representation that enables building translation equivariant models; and (3) a reconstruction loss tailored to capture overall object geometry. We reconstruct all objects jointly in one pass, producing a coherent reconstruction, where all objects live in a single consistent 3D coordinate frame relative to the camera and they do not intersect in 3D space.
  arXiv  Detail & Related papers  (2020-04-27T17:53:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.