Geometry Aware Field-to-field Transformations for 3D Semantic Segmentation

• URL: http://arxiv.org/abs/2310.05133v1
• Date: Sun, 8 Oct 2023 11:48:19 GMT
• Title: Geometry Aware Field-to-field Transformations for 3D Semantic Segmentation
• Authors: Dominik Hollidt, Clinton Wang, Polina Golland, Marc Pollefeys
• Abstract summary: We present a novel approach to perform 3D semantic segmentation solely from 2D supervision by leveraging Neural Radiance Fields (NeRFs) By extracting features along a surface point cloud, we achieve a compact representation of the scene which is sample-efficient and conducive to 3D reasoning.
• Score: 48.307734886370014
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present a novel approach to perform 3D semantic segmentation solely from 2D supervision by leveraging Neural Radiance Fields (NeRFs). By extracting features along a surface point cloud, we achieve a compact representation of the scene which is sample-efficient and conducive to 3D reasoning. Learning this feature space in an unsupervised manner via masked autoencoding enables few-shot segmentation. Our method is agnostic to the scene parameterization, working on scenes fit with any type of NeRF.

Related papers

• Neural Attention Field: Emerging Point Relevance in 3D Scenes for One-Shot Dexterous Grasping [34.98831146003579]
  One-shot transfer of dexterous grasps to novel scenes with object and context variations has been a challenging problem. We propose the textitneural attention field for representing semantic-aware dense feature fields in the 3D space.
  arXiv  Detail & Related papers  (2024-10-30T14:06:51Z)
• PanopticRecon: Leverage Open-vocabulary Instance Segmentation for Zero-shot Panoptic Reconstruction [23.798691661418253]
  We propose a novel zero-shot panoptic reconstruction method from RGB-D images of scenes. We tackle both challenges by propagating partial labels with the aid of dense generalized features. Our method outperforms state-of-the-art methods on the indoor dataset ScanNet V2 and the outdoor dataset KITTI-360.
  arXiv  Detail & Related papers  (2024-07-01T15:06:04Z)
• View-Consistent Hierarchical 3D Segmentation Using Ultrametric Feature Fields [52.08335264414515]
  We learn a novel feature field within a Neural Radiance Field (NeRF) representing a 3D scene. Our method takes view-inconsistent multi-granularity 2D segmentations as input and produces a hierarchy of 3D-consistent segmentations as output. We evaluate our method and several baselines on synthetic datasets with multi-view images and multi-granular segmentation, showcasing improved accuracy and viewpoint-consistency.
  arXiv  Detail & Related papers  (2024-05-30T04:14:58Z)
• Scene-Generalizable Interactive Segmentation of Radiance Fields [64.37093918762]
  We make the first attempt at Scene-Generalizable Interactive in Radiance Fields (SGISRF) We propose a novel SGISRF method, which can perform 3D object segmentation for novel (unseen) scenes represented by radiance fields, guided by only a few interactive user clicks in a given set of multi-view 2D images. Experiments on two real-world challenging benchmarks covering diverse scenes demonstrate 1) effectiveness and scene-generalizability of the proposed method, 2) favorable performance compared to classical method requiring scene-specific optimization.
  arXiv  Detail & Related papers  (2023-08-09T17:55:50Z)
• ONeRF: Unsupervised 3D Object Segmentation from Multiple Views [59.445957699136564]
  ONeRF is a method that automatically segments and reconstructs object instances in 3D from multi-view RGB images without any additional manual annotations. The segmented 3D objects are represented using separate Neural Radiance Fields (NeRFs) which allow for various 3D scene editing and novel view rendering.
  arXiv  Detail & Related papers  (2022-11-22T06:19:37Z)
• Unsupervised Multi-View Object Segmentation Using Radiance Field Propagation [55.9577535403381]
  We present a novel approach to segmenting objects in 3D during reconstruction given only unlabeled multi-view images of a scene. The core of our method is a novel propagation strategy for individual objects' radiance fields with a bidirectional photometric loss. To the best of our knowledge, RFP is the first unsupervised approach for tackling 3D scene object segmentation for neural radiance field (NeRF)
  arXiv  Detail & Related papers  (2022-10-02T11:14:23Z)
• Cylinder3D: An Effective 3D Framework for Driving-scene LiDAR Semantic Segmentation [87.54570024320354]
  State-of-the-art methods for large-scale driving-scene LiDAR semantic segmentation often project and process the point clouds in the 2D space. A straightforward solution to tackle the issue of 3D-to-2D projection is to keep the 3D representation and process the points in the 3D space. We develop a 3D cylinder partition and a 3D cylinder convolution based framework, termed as Cylinder3D, which exploits the 3D topology relations and structures of driving-scene point clouds.
  arXiv  Detail & Related papers  (2020-08-04T13:56:19Z)
• Semantic Implicit Neural Scene Representations With Semi-Supervised Training [47.61092265963234]
  We show that implicit neural scene representations can be leveraged to perform per-point semantic segmentation. Our method is simple, general, and only requires a few tens of labeled 2D segmentation masks. We explore two novel applications for this semantically aware implicit neural scene representation.
  arXiv  Detail & Related papers  (2020-03-28T00:43:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.