GO-NeRF: Generating Virtual Objects in Neural Radiance Fields

• URL: http://arxiv.org/abs/2401.05750v1
• Date: Thu, 11 Jan 2024 08:58:13 GMT
• Title: GO-NeRF: Generating Virtual Objects in Neural Radiance Fields
• Authors: Peng Dai and Feitong Tan and Xin Yu and Yinda Zhang and Xiaojuan Qi
• Abstract summary: GO-NeRF is capable of utilizing scene context for high-quality and harmonious 3D object generation within an existing NeRF. Our method employs a compositional rendering formulation that allows the generated 3D objects to be seamlessly composited into the scene.
• Score: 75.13534508391852
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Despite advances in 3D generation, the direct creation of 3D objects within an existing 3D scene represented as NeRF remains underexplored. This process requires not only high-quality 3D object generation but also seamless composition of the generated 3D content into the existing NeRF. To this end, we propose a new method, GO-NeRF, capable of utilizing scene context for high-quality and harmonious 3D object generation within an existing NeRF. Our method employs a compositional rendering formulation that allows the generated 3D objects to be seamlessly composited into the scene utilizing learned 3D-aware opacity maps without introducing unintended scene modification. Moreover, we also develop tailored optimization objectives and training strategies to enhance the model's ability to exploit scene context and mitigate artifacts, such as floaters, originating from 3D object generation within a scene. Extensive experiments on both feed-forward and $360^o$ scenes show the superior performance of our proposed GO-NeRF in generating objects harmoniously composited with surrounding scenes and synthesizing high-quality novel view images. Project page at {\url{https://daipengwa.github.io/GO-NeRF/}.

Related papers

• ScalingGaussian: Enhancing 3D Content Creation with Generative Gaussian Splatting [30.99112626706754]
  The creation of high-quality 3D assets is paramount for applications in digital heritage, entertainment, and robotics. Traditionally, this process necessitates skilled professionals and specialized software for modeling. We introduce a novel 3D content creation framework, which generates 3D textures efficiently.
  arXiv  Detail & Related papers  (2024-07-26T18:26:01Z)
• 3D-SceneDreamer: Text-Driven 3D-Consistent Scene Generation [51.64796781728106]
  We propose a generative refinement network to synthesize new contents with higher quality by exploiting the natural image prior to 2D diffusion model and the global 3D information of the current scene. Our approach supports wide variety of scene generation and arbitrary camera trajectories with improved visual quality and 3D consistency.
  arXiv  Detail & Related papers  (2024-03-14T14:31:22Z)
• Denoising Diffusion via Image-Based Rendering [54.20828696348574]
  We introduce the first diffusion model able to perform fast, detailed reconstruction and generation of real-world 3D scenes. First, we introduce a new neural scene representation, IB-planes, that can efficiently and accurately represent large 3D scenes. Second, we propose a denoising-diffusion framework to learn a prior over this novel 3D scene representation, using only 2D images.
  arXiv  Detail & Related papers  (2024-02-05T19:00:45Z)
• Blended-NeRF: Zero-Shot Object Generation and Blending in Existing Neural Radiance Fields [26.85599376826124]
  We present Blended-NeRF, a framework for editing a specific region of interest in an existing NeRF scene. We allow local editing by localizing a 3D ROI box in the input scene, and blend the content synthesized inside the ROI with the existing scene. We show our framework for several 3D editing applications, including adding new objects to a scene, removing/altering existing objects, and texture conversion.
  arXiv  Detail & Related papers  (2023-06-22T09:34:55Z)
• DITTO-NeRF: Diffusion-based Iterative Text To Omni-directional 3D Model [15.091263190886337]
  We propose a novel pipeline to generate a high-quality 3D NeRF model from a text prompt or a single image. DitTO-NeRF consists of constructing high-quality partial 3D object for limited in-boundary (IB) angles using the given or text-generated 2D image from the frontal view. We propose progressive 3D object reconstruction schemes in terms of scales (low to high resolution), angles (IB angles initially to outer-boundary (OB) later, and masks (object to background boundary) in our DITTO-NeRF.
  arXiv  Detail & Related papers  (2023-04-06T02:27:22Z)
• NeRF-GAN Distillation for Efficient 3D-Aware Generation with Convolutions [97.27105725738016]
  integration of Neural Radiance Fields (NeRFs) and generative models, such as Generative Adversarial Networks (GANs) has transformed 3D-aware generation from single-view images. We propose a simple and effective method, based on re-using the well-disentangled latent space of a pre-trained NeRF-GAN in a pose-conditioned convolutional network to directly generate 3D-consistent images corresponding to the underlying 3D representations.
  arXiv  Detail & Related papers  (2023-03-22T18:59:48Z)
• CC3D: Layout-Conditioned Generation of Compositional 3D Scenes [49.281006972028194]
  We introduce CC3D, a conditional generative model that synthesizes complex 3D scenes conditioned on 2D semantic scene layouts. Our evaluations on synthetic 3D-FRONT and real-world KITTI-360 datasets demonstrate that our model generates scenes of improved visual and geometric quality.
  arXiv  Detail & Related papers  (2023-03-21T17:59:02Z)
• 3D-aware Image Synthesis via Learning Structural and Textural Representations [39.681030539374994]
  We propose VolumeGAN, for high-fidelity 3D-aware image synthesis, through explicitly learning a structural representation and a textural representation. Our approach achieves sufficiently higher image quality and better 3D control than the previous methods.
  arXiv  Detail & Related papers  (2021-12-20T18:59:40Z)
• Towards Realistic 3D Embedding via View Alignment [53.89445873577063]
  This paper presents an innovative View Alignment GAN (VA-GAN) that composes new images by embedding 3D models into 2D background images realistically and automatically. VA-GAN consists of a texture generator and a differential discriminator that are inter-connected and end-to-end trainable.
  arXiv  Detail & Related papers  (2020-07-14T14:45:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.