MOSE: Monocular Semantic Reconstruction Using NeRF-Lifted Noisy Priors

• URL: http://arxiv.org/abs/2409.14019v1
• Date: Sat, 21 Sep 2024 05:12:13 GMT
• Title: MOSE: Monocular Semantic Reconstruction Using NeRF-Lifted Noisy Priors
• Authors: Zhenhua Du, Binbin Xu, Haoyu Zhang, Kai Huo, Shuaifeng Zhi,
• Abstract summary: We propose a neural field semantic reconstruction approach to lift inferred image-level noisy priors to 3D. Our method produces accurate semantics and geometry in both 3D and 2D space.
• Score: 11.118490283303407
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Accurately reconstructing dense and semantically annotated 3D meshes from monocular images remains a challenging task due to the lack of geometry guidance and imperfect view-dependent 2D priors. Though we have witnessed recent advancements in implicit neural scene representations enabling precise 2D rendering simply from multi-view images, there have been few works addressing 3D scene understanding with monocular priors alone. In this paper, we propose MOSE, a neural field semantic reconstruction approach to lift inferred image-level noisy priors to 3D, producing accurate semantics and geometry in both 3D and 2D space. The key motivation for our method is to leverage generic class-agnostic segment masks as guidance to promote local consistency of rendered semantics during training. With the help of semantics, we further apply a smoothness regularization to texture-less regions for better geometric quality, thus achieving mutual benefits of geometry and semantics. Experiments on the ScanNet dataset show that our MOSE outperforms relevant baselines across all metrics on tasks of 3D semantic segmentation, 2D semantic segmentation and 3D surface reconstruction.

Related papers

• Large Spatial Model: End-to-end Unposed Images to Semantic 3D [79.94479633598102]
  Large Spatial Model (LSM) processes unposed RGB images directly into semantic radiance fields. LSM simultaneously estimates geometry, appearance, and semantics in a single feed-forward operation. It can generate versatile label maps by interacting with language at novel viewpoints.
  arXiv  Detail & Related papers  (2024-10-24T17:54:42Z)
• Mask3D: Pre-training 2D Vision Transformers by Learning Masked 3D Priors [29.419069066603438]
  We propose Mask3D to leverage existing large-scale RGB-D data in a self-supervised pre-training to embed 3D priors into 2D learned feature representations. We demonstrate Mask3D is particularly effective in embedding 3D priors into the powerful 2D ViT backbone, enabling improved representation learning for various scene understanding tasks.
  arXiv  Detail & Related papers  (2023-02-28T16:45:21Z)
• SSR-2D: Semantic 3D Scene Reconstruction from 2D Images [54.46126685716471]
  In this work, we explore a central 3D scene modeling task, namely, semantic scene reconstruction without using any 3D annotations. The key idea of our approach is to design a trainable model that employs both incomplete 3D reconstructions and their corresponding source RGB-D images. Our method achieves the state-of-the-art performance of semantic scene completion on two large-scale benchmark datasets MatterPort3D and ScanNet.
  arXiv  Detail & Related papers  (2023-02-07T17:47:52Z)
• MvDeCor: Multi-view Dense Correspondence Learning for Fine-grained 3D Segmentation [91.6658845016214]
  We propose to utilize self-supervised techniques in the 2D domain for fine-grained 3D shape segmentation tasks. We render a 3D shape from multiple views, and set up a dense correspondence learning task within the contrastive learning framework. As a result, the learned 2D representations are view-invariant and geometrically consistent.
  arXiv  Detail & Related papers  (2022-08-18T00:48:15Z)
• Monocular 3D Object Reconstruction with GAN Inversion [122.96094885939146]
  MeshInversion is a novel framework to improve the reconstruction of textured 3D meshes. It exploits the generative prior of a 3D GAN pre-trained for 3D textured mesh synthesis. Our framework obtains faithful 3D reconstructions with consistent geometry and texture across both observed and unobserved parts.
  arXiv  Detail & Related papers  (2022-07-20T17:47:22Z)
• Self-Supervised Image Representation Learning with Geometric Set Consistency [50.12720780102395]
  We propose a method for self-supervised image representation learning under the guidance of 3D geometric consistency. Specifically, we introduce 3D geometric consistency into a contrastive learning framework to enforce the feature consistency within image views.
  arXiv  Detail & Related papers  (2022-03-29T08:57:33Z)
• 3D Shape Reconstruction from 2D Images with Disentangled Attribute Flow [61.62796058294777]
  Reconstructing 3D shape from a single 2D image is a challenging task. Most of the previous methods still struggle to extract semantic attributes for 3D reconstruction task. We propose 3DAttriFlow to disentangle and extract semantic attributes through different semantic levels in the input images.
  arXiv  Detail & Related papers  (2022-03-29T02:03:31Z)
• NeSF: Neural Semantic Fields for Generalizable Semantic Segmentation of 3D Scenes [25.26518805603798]
  NeSF is a method for producing 3D semantic fields from posed RGB images alone. Our method is the first to offer truly dense 3D scene segmentations requiring only 2D supervision for training.
  arXiv  Detail & Related papers  (2021-11-25T21:44:54Z)
• Semantic Implicit Neural Scene Representations With Semi-Supervised Training [47.61092265963234]
  We show that implicit neural scene representations can be leveraged to perform per-point semantic segmentation. Our method is simple, general, and only requires a few tens of labeled 2D segmentation masks. We explore two novel applications for this semantically aware implicit neural scene representation.
  arXiv  Detail & Related papers  (2020-03-28T00:43:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.