SANeRF-HQ: Segment Anything for NeRF in High Quality

• URL: http://arxiv.org/abs/2312.01531v2
• Date: Sat, 6 Apr 2024 07:04:29 GMT
• Title: SANeRF-HQ: Segment Anything for NeRF in High Quality
• Authors: Yichen Liu, Benran Hu, Chi-Keung Tang, Yu-Wing Tai,
• Abstract summary: We introduce the Segment Anything for NeRF in High Quality (SANeRF-HQ) to achieve high-quality 3D segmentation of any target object in a given scene. We employ density field and RGB similarity to enhance the accuracy of segmentation boundary during the aggregation.
• Score: 61.77762568224097
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recently, the Segment Anything Model (SAM) has showcased remarkable capabilities of zero-shot segmentation, while NeRF (Neural Radiance Fields) has gained popularity as a method for various 3D problems beyond novel view synthesis. Though there exist initial attempts to incorporate these two methods into 3D segmentation, they face the challenge of accurately and consistently segmenting objects in complex scenarios. In this paper, we introduce the Segment Anything for NeRF in High Quality (SANeRF-HQ) to achieve high-quality 3D segmentation of any target object in a given scene. SANeRF-HQ utilizes SAM for open-world object segmentation guided by user-supplied prompts, while leveraging NeRF to aggregate information from different viewpoints. To overcome the aforementioned challenges, we employ density field and RGB similarity to enhance the accuracy of segmentation boundary during the aggregation. Emphasizing on segmentation accuracy, we evaluate our method on multiple NeRF datasets where high-quality ground-truths are available or manually annotated. SANeRF-HQ shows a significant quality improvement over state-of-the-art methods in NeRF object segmentation, provides higher flexibility for object localization, and enables more consistent object segmentation across multiple views. Results and code are available at the project site: https://lyclyc52.github.io/SANeRF-HQ/.

Related papers

• Explicit-NeRF-QA: A Quality Assessment Database for Explicit NeRF Model Compression [10.469092315640696]
  We construct a new dataset, called Explicit-NeRF-QA, to address the challenge of the NeRF compression study. We use 22 3D objects with diverse geometries, textures, and material complexities to train four typical explicit NeRF models. A subjective experiment with lab environment is conducted to collect subjective scores from 21 viewers.
  arXiv  Detail & Related papers  (2024-07-11T04:02:05Z)
• View-Consistent Hierarchical 3D Segmentation Using Ultrametric Feature Fields [52.08335264414515]
  We learn a novel feature field within a Neural Radiance Field (NeRF) representing a 3D scene. Our method takes view-inconsistent multi-granularity 2D segmentations as input and produces a hierarchy of 3D-consistent segmentations as output. We evaluate our method and several baselines on synthetic datasets with multi-view images and multi-granular segmentation, showcasing improved accuracy and viewpoint-consistency.
  arXiv  Detail & Related papers  (2024-05-30T04:14:58Z)
• SAID-NeRF: Segmentation-AIDed NeRF for Depth Completion of Transparent Objects [7.529049797077149]
  Acquiring accurate depth information of transparent objects using off-the-shelf RGB-D cameras is a well-known challenge in Computer Vision and Robotics. NeRFs are learning-free approaches and have demonstrated wide success in novel view synthesis and shape recovery. Our proposed method-AID-NeRF shows significant performance on depth completion datasets for transparent objects and robotic grasping.
  arXiv  Detail & Related papers  (2024-03-28T17:28:32Z)
• Obj-NeRF: Extract Object NeRFs from Multi-view Images [7.669778218573394]
  We propose -NeRF, a comprehensive pipeline that recovers the 3D geometry of a specific object from multi-view images using a single prompt. We also apply -NeRF to various applications, including object removal, rotation, replacement, and recoloring.
  arXiv  Detail & Related papers  (2023-11-26T13:15:37Z)
• GP-NeRF: Generalized Perception NeRF for Context-Aware 3D Scene Understanding [101.32590239809113]
  Generalized Perception NeRF (GP-NeRF) is a novel pipeline that makes the widely used segmentation model and NeRF work compatibly under a unified framework. We propose two self-distillation mechanisms, i.e., the Semantic Distill Loss and the Depth-Guided Semantic Distill Loss, to enhance the discrimination and quality of the semantic field.
  arXiv  Detail & Related papers  (2023-11-20T15:59:41Z)
• Instance Neural Radiance Field [62.152611795824185]
  This paper presents one of the first learning-based NeRF 3D instance segmentation pipelines, dubbed as Instance Neural Radiance Field. We adopt a 3D proposal-based mask prediction network on the sampled volumetric features from NeRF. Our method is also one of the first to achieve such results in pure inference.
  arXiv  Detail & Related papers  (2023-04-10T05:49:24Z)
• SegNeRF: 3D Part Segmentation with Neural Radiance Fields [63.12841224024818]
  SegNeRF is a neural field representation that integrates a semantic field along with the usual radiance field. SegNeRF is capable of simultaneously predicting geometry, appearance, and semantic information from posed images, even for unseen objects. SegNeRF is able to generate an explicit 3D model from a single image of an object taken in the wild, with its corresponding part segmentation.
  arXiv  Detail & Related papers  (2022-11-21T07:16:03Z)
• NeRF-SOS: Any-View Self-supervised Object Segmentation from Complex Real-World Scenes [80.59831861186227]
  This paper carries out the exploration of self-supervised learning for object segmentation using NeRF for complex real-world scenes. Our framework, called NeRF with Self-supervised Object NeRF-SOS, encourages NeRF models to distill compact geometry-aware segmentation clusters. It consistently surpasses other 2D-based self-supervised baselines and predicts finer semantics masks than existing supervised counterparts.
  arXiv  Detail & Related papers  (2022-09-19T06:03:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.