ObjectCarver: Semi-automatic segmentation, reconstruction and separation of 3D objects

• URL: http://arxiv.org/abs/2407.19108v1
• Date: Fri, 26 Jul 2024 22:13:20 GMT
• Title: ObjectCarver: Semi-automatic segmentation, reconstruction and separation of 3D objects
• Authors: Gemmechu Hassena, Jonathan Moon, Ryan Fujii, Andrew Yuen, Noah Snavely, Steve Marschner, Bharath Hariharan,
• Abstract summary: Implicit neural fields have made remarkable progress in reconstructing 3D surfaces from multiple images. Previous work has attempted to tackle this problem by introducing a framework to train separate signed distance fields. We introduce our method, ObjectCarver, to tackle the problem of object separation from just click input in a single view.
• Score: 44.38881095466177
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Implicit neural fields have made remarkable progress in reconstructing 3D surfaces from multiple images; however, they encounter challenges when it comes to separating individual objects within a scene. Previous work has attempted to tackle this problem by introducing a framework to train separate signed distance fields (SDFs) simultaneously for each of N objects and using a regularization term to prevent objects from overlapping. However, all of these methods require segmentation masks to be provided, which are not always readily available. We introduce our method, ObjectCarver, to tackle the problem of object separation from just click input in a single view. Given posed multi-view images and a set of user-input clicks to prompt segmentation of the individual objects, our method decomposes the scene into separate objects and reconstructs a high-quality 3D surface for each one. We introduce a loss function that prevents floaters and avoids inappropriate carving-out due to occlusion. In addition, we introduce a novel scene initialization method that significantly speeds up the process while preserving geometric details compared to previous approaches. Despite requiring neither ground truth masks nor monocular cues, our method outperforms baselines both qualitatively and quantitatively. In addition, we introduce a new benchmark dataset for evaluation.

Related papers

• OR-NeRF: Object Removing from 3D Scenes Guided by Multiview Segmentation with Neural Radiance Fields [53.32527220134249]
  The emergence of Neural Radiance Fields (NeRF) for novel view synthesis has increased interest in 3D scene editing. Current methods face challenges such as time-consuming object labeling, limited capability to remove specific targets, and compromised rendering quality after removal. This paper proposes a novel object-removing pipeline, named OR-NeRF, that can remove objects from 3D scenes with user-given points or text prompts on a single view.
  arXiv  Detail & Related papers  (2023-05-17T18:18:05Z)
• BundleSDF: Neural 6-DoF Tracking and 3D Reconstruction of Unknown Objects [89.2314092102403]
  We present a near real-time method for 6-DoF tracking of an unknown object from a monocular RGBD video sequence. Our method works for arbitrary rigid objects, even when visual texture is largely absent.
  arXiv  Detail & Related papers  (2023-03-24T17:13:49Z)
• OGC: Unsupervised 3D Object Segmentation from Rigid Dynamics of Point Clouds [4.709764624933227]
  We propose the first unsupervised method, called OGC, to simultaneously identify multiple 3D objects in a single forward pass. We extensively evaluate our method on five datasets, demonstrating the superior performance for object part instance segmentation.
  arXiv  Detail & Related papers  (2022-10-10T07:01:08Z)
• Unsupervised Multi-View Object Segmentation Using Radiance Field Propagation [55.9577535403381]
  We present a novel approach to segmenting objects in 3D during reconstruction given only unlabeled multi-view images of a scene. The core of our method is a novel propagation strategy for individual objects' radiance fields with a bidirectional photometric loss. To the best of our knowledge, RFP is the first unsupervised approach for tackling 3D scene object segmentation for neural radiance field (NeRF)
  arXiv  Detail & Related papers  (2022-10-02T11:14:23Z)
• Objects are Different: Flexible Monocular 3D Object Detection [87.82253067302561]
  We propose a flexible framework for monocular 3D object detection which explicitly decouples the truncated objects and adaptively combines multiple approaches for object depth estimation. Experiments demonstrate that our method outperforms the state-of-the-art method by relatively 27% for the moderate level and 30% for the hard level in the test set of KITTI benchmark.
  arXiv  Detail & Related papers  (2021-04-06T07:01:28Z)
• Holistic 3D Scene Understanding from a Single Image with Implicit Representation [112.40630836979273]
  We present a new pipeline for holistic 3D scene understanding from a single image. We propose an image-based local structured implicit network to improve the object shape estimation. We also refine 3D object pose and scene layout via a novel implicit scene graph neural network.
  arXiv  Detail & Related papers  (2021-03-11T02:52:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.