FiG-NeRF: Figure-Ground Neural Radiance Fields for 3D Object Category Modelling

• URL: http://arxiv.org/abs/2104.08418v1
• Date: Sat, 17 Apr 2021 01:38:54 GMT
• Title: FiG-NeRF: Figure-Ground Neural Radiance Fields for 3D Object Category Modelling
• Authors: Christopher Xie, Keunhong Park, Ricardo Martin-Brualla, Matthew Brown
• Abstract summary: We use Neural Radiance Fields (NeRF) to learn high quality 3D object category models from collections of input images. We show that this method can learn accurate 3D object category models using only photometric supervision and casually captured images.
• Score: 11.432178728985956
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We investigate the use of Neural Radiance Fields (NeRF) to learn high quality 3D object category models from collections of input images. In contrast to previous work, we are able to do this whilst simultaneously separating foreground objects from their varying backgrounds. We achieve this via a 2-component NeRF model, FiG-NeRF, that prefers explanation of the scene as a geometrically constant background and a deformable foreground that represents the object category. We show that this method can learn accurate 3D object category models using only photometric supervision and casually captured images of the objects. Additionally, our 2-part decomposition allows the model to perform accurate and crisp amodal segmentation. We quantitatively evaluate our method with view synthesis and image fidelity metrics, using synthetic, lab-captured, and in-the-wild data. Our results demonstrate convincing 3D object category modelling that exceed the performance of existing methods.

Related papers

• DistillNeRF: Perceiving 3D Scenes from Single-Glance Images by Distilling Neural Fields and Foundation Model Features [65.8738034806085]
  DistillNeRF is a self-supervised learning framework for understanding 3D environments in autonomous driving scenes. Our method is a generalizable feedforward model that predicts a rich neural scene representation from sparse, single-frame multi-view camera inputs.
  arXiv  Detail & Related papers  (2024-06-17T21:15:13Z)
• Learning 3D-Aware GANs from Unposed Images with Template Feature Field [33.32761749864555]
  This work targets learning 3D-aware GANs from unposed images. We propose to perform on-the-fly pose estimation of training images with a learned template feature field (TeFF)
  arXiv  Detail & Related papers  (2024-04-08T17:42:08Z)
• ComboVerse: Compositional 3D Assets Creation Using Spatially-Aware Diffusion Guidance [76.7746870349809]
  We present ComboVerse, a 3D generation framework that produces high-quality 3D assets with complex compositions by learning to combine multiple models. Our proposed framework emphasizes spatial alignment of objects, compared with standard score distillation sampling.
  arXiv  Detail & Related papers  (2024-03-19T03:39:43Z)
• Likelihood-Based Generative Radiance Field with Latent Space Energy-Based Model for 3D-Aware Disentangled Image Representation [43.41596483002523]
  We propose a likelihood-based top-down 3D-aware 2D image generative model that incorporates 3D representation via Neural Radiance Fields (NeRF) and 2D imaging process via differentiable volume rendering. Experiments on several benchmark datasets demonstrate that the NeRF-LEBM can infer 3D object structures from 2D images, generate 2D images with novel views and objects, learn from incomplete 2D images, and learn from 2D images with known or unknown camera poses.
  arXiv  Detail & Related papers  (2023-04-16T23:44:41Z)
• MegaPose: 6D Pose Estimation of Novel Objects via Render & Compare [84.80956484848505]
  MegaPose is a method to estimate the 6D pose of novel objects, that is, objects unseen during training. We present a 6D pose refiner based on a render&compare strategy which can be applied to novel objects. Second, we introduce a novel approach for coarse pose estimation which leverages a network trained to classify whether the pose error between a synthetic rendering and an observed image of the same object can be corrected by the refiner.
  arXiv  Detail & Related papers  (2022-12-13T19:30:03Z)
• Generative Deformable Radiance Fields for Disentangled Image Synthesis of Topology-Varying Objects [52.46838926521572]
  3D-aware generative models have demonstrated their superb performance to generate 3D neural radiance fields (NeRF) from a collection of monocular 2D images. We propose a generative model for synthesizing radiance fields of topology-varying objects with disentangled shape and appearance variations.
  arXiv  Detail & Related papers  (2022-09-09T08:44:06Z)
• GAN2X: Non-Lambertian Inverse Rendering of Image GANs [85.76426471872855]
  We present GAN2X, a new method for unsupervised inverse rendering that only uses unpaired images for training. Unlike previous Shape-from-GAN approaches that mainly focus on 3D shapes, we take the first attempt to also recover non-Lambertian material properties by exploiting the pseudo paired data generated by a GAN. Experiments demonstrate that GAN2X can accurately decompose 2D images to 3D shape, albedo, and specular properties for different object categories, and achieves the state-of-the-art performance for unsupervised single-view 3D face reconstruction.
  arXiv  Detail & Related papers  (2022-06-18T16:58:49Z)
• Template NeRF: Towards Modeling Dense Shape Correspondences from Category-Specific Object Images [4.662583832063716]
  We present neural radiance fields (NeRF) with templates, dubbed template-NeRF, for modeling appearance and geometry. We generate dense shape correspondences simultaneously among objects of the same category from only multi-view posed images. The learned dense correspondences can be readily used for various image-based tasks such as keypoint detection, part segmentation, and texture transfer.
  arXiv  Detail & Related papers  (2021-11-08T02:16:48Z)
• Aug3D-RPN: Improving Monocular 3D Object Detection by Synthetic Images with Virtual Depth [64.29043589521308]
  We propose a rendering module to augment the training data by synthesizing images with virtual-depths. The rendering module takes as input the RGB image and its corresponding sparse depth image, outputs a variety of photo-realistic synthetic images. Besides, we introduce an auxiliary module to improve the detection model by jointly optimizing it through a depth estimation task.
  arXiv  Detail & Related papers  (2021-07-28T11:00:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.