DORec: Decomposed Object Reconstruction and Segmentation Utilizing 2D Self-Supervised Features

• URL: http://arxiv.org/abs/2310.11092v3
• Date: Mon, 2 Sep 2024 08:37:29 GMT
• Title: DORec: Decomposed Object Reconstruction and Segmentation Utilizing 2D Self-Supervised Features
• Authors: Jun Wu, Sicheng Li, Sihui Ji, Yifei Yang, Yue Wang, Rong Xiong, Yiyi Liao,
• Abstract summary: We propose a Decomposed Object Reconstruction network based on neural implicit representations. Our key idea is to use 2D self-supervised features to create two levels of masks for supervision: a binary mask for foreground regions and a K-cluster mask for semantically similar regions. Experimental results on different datasets show DORec's superiority in segmenting and reconstructing diverse foreground objects from varied backgrounds enabling downstream tasks such as pose estimation.
• Score: 33.04128557550782
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recovering 3D geometry and textures of individual objects is crucial for many robotics applications, such as manipulation, pose estimation, and autonomous driving. However, decomposing a target object from a complex background is challenging. Most existing approaches rely on costly manual labels to acquire object instance perception. Recent advancements in 2D self-supervised learning offer new prospects for identifying objects of interest, yet leveraging such noisy 2D features for clean decomposition remains difficult. In this paper, we propose a Decomposed Object Reconstruction (DORec) network based on neural implicit representations. Our key idea is to use 2D self-supervised features to create two levels of masks for supervision: a binary mask for foreground regions and a K-cluster mask for semantically similar regions. These complementary masks result in robust decomposition. Experimental results on different datasets show DORec's superiority in segmenting and reconstructing diverse foreground objects from varied backgrounds enabling downstream tasks such as pose estimation.

Related papers

• PickScan: Object discovery and reconstruction from handheld interactions [99.99566882133179]
  We develop an interaction-guided and class-agnostic method to reconstruct 3D representations of scenes. Our main contribution is a novel approach to detecting user-object interactions and extracting the masks of manipulated objects. Compared to Co-Fusion, the only comparable interaction-based and class-agnostic baseline, this corresponds to a reduction in chamfer distance of 73%.
  arXiv  Detail & Related papers  (2024-11-17T23:09:08Z)
• DiscoNeRF: Class-Agnostic Object Field for 3D Object Discovery [46.711276257688326]
  NeRFs have become a powerful tool for modeling 3D scenes from multiple images. Previous approaches to 3D segmentation of NeRFs either require user interaction to isolate a single object, or they rely on 2D semantic masks with a limited number of classes for supervision. We propose a method that is robust to inconsistent segmentations and successfully decomposes the scene into a set of objects of any class.
  arXiv  Detail & Related papers  (2024-08-19T12:07:24Z)
• MonoMAE: Enhancing Monocular 3D Detection through Depth-Aware Masked Autoencoders [93.87585467898252]
  We design MonoMAE, a monocular 3D detector inspired by Masked Autoencoders. MonoMAE consists of two novel designs. The first is depth-aware masking that selectively masks certain parts of non-occluded object queries. The second is lightweight query completion that works with the depth-aware masking to learn to reconstruct and complete the masked object queries.
  arXiv  Detail & Related papers  (2024-05-13T12:32:45Z)
• SUGAR: Pre-training 3D Visual Representations for Robotics [85.55534363501131]
  We introduce a novel 3D pre-training framework for robotics named SUGAR. SUGAR captures semantic, geometric and affordance properties of objects through 3D point clouds. We show that SUGAR's 3D representation outperforms state-of-the-art 2D and 3D representations.
  arXiv  Detail & Related papers  (2024-04-01T21:23:03Z)
• Iterative Superquadric Recomposition of 3D Objects from Multiple Views [77.53142165205283]
  We propose a framework, ISCO, to recompose an object using 3D superquadrics as semantic parts directly from 2D views. Our framework iteratively adds new superquadrics wherever the reconstruction error is high. It provides consistently more accurate 3D reconstructions, even from images in the wild.
  arXiv  Detail & Related papers  (2023-09-05T10:21:37Z)
• AutoRecon: Automated 3D Object Discovery and Reconstruction [41.60050228813979]
  We propose a novel framework named AutoRecon for the automated discovery and reconstruction of an object from multi-view images. We demonstrate that foreground objects can be robustly located and segmented from SfM point clouds by leveraging self-supervised 2D vision transformer features. Experiments on the DTU, BlendedMVS and CO3D-V2 datasets demonstrate the effectiveness and robustness of AutoRecon.
  arXiv  Detail & Related papers  (2023-05-15T17:16:46Z)
• CASAPose: Class-Adaptive and Semantic-Aware Multi-Object Pose Estimation [2.861848675707602]
  We present a new single-stage architecture called CASAPose. It determines 2D-3D correspondences for pose estimation of multiple different objects in RGB images in one pass. It is fast and memory efficient, and achieves high accuracy for multiple objects.
  arXiv  Detail & Related papers  (2022-10-11T10:20:01Z)
• OGC: Unsupervised 3D Object Segmentation from Rigid Dynamics of Point Clouds [4.709764624933227]
  We propose the first unsupervised method, called OGC, to simultaneously identify multiple 3D objects in a single forward pass. We extensively evaluate our method on five datasets, demonstrating the superior performance for object part instance segmentation.
  arXiv  Detail & Related papers  (2022-10-10T07:01:08Z)
• Unsupervised Learning of 3D Object Categories from Videos in the Wild [75.09720013151247]
  We focus on learning a model from multiple views of a large collection of object instances. We propose a new neural network design, called warp-conditioned ray embedding (WCR), which significantly improves reconstruction. Our evaluation demonstrates performance improvements over several deep monocular reconstruction baselines on existing benchmarks.
  arXiv  Detail & Related papers  (2021-03-30T17:57:01Z)
• AutoSweep: Recovering 3D Editable Objectsfrom a Single Photograph [54.701098964773756]
  We aim to recover 3D objects with semantic parts and can be directly edited. Our work makes an attempt towards recovering two types of primitive-shaped objects, namely, generalized cuboids and generalized cylinders. Our algorithm can recover high quality 3D models and outperforms existing methods in both instance segmentation and 3D reconstruction.
  arXiv  Detail & Related papers  (2020-05-27T12:16:24Z)
• SDOD:Real-time Segmenting and Detecting 3D Object by Depth [5.97602869680438]
  This paper proposes a real-time framework that segmenting and detecting 3D objects by depth. We discretize the objects' depth into depth categories and transform the instance segmentation task into a pixel-level classification task. Experiments on the challenging KITTI dataset show that our approach outperforms LklNet about 1.8 times on the speed of segmentation and 3D detection.
  arXiv  Detail & Related papers  (2020-01-26T09:06:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.