Blended-NeRF: Zero-Shot Object Generation and Blending in Existing Neural Radiance Fields

• URL: http://arxiv.org/abs/2306.12760v2
• Date: Thu, 7 Sep 2023 10:30:10 GMT
• Title: Blended-NeRF: Zero-Shot Object Generation and Blending in Existing Neural Radiance Fields
• Authors: Ori Gordon and Omri Avrahami and Dani Lischinski
• Abstract summary: We present Blended-NeRF, a framework for editing a specific region of interest in an existing NeRF scene. We allow local editing by localizing a 3D ROI box in the input scene, and blend the content synthesized inside the ROI with the existing scene. We show our framework for several 3D editing applications, including adding new objects to a scene, removing/altering existing objects, and texture conversion.
• Score: 26.85599376826124
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Editing a local region or a specific object in a 3D scene represented by a NeRF or consistently blending a new realistic object into the scene is challenging, mainly due to the implicit nature of the scene representation. We present Blended-NeRF, a robust and flexible framework for editing a specific region of interest in an existing NeRF scene, based on text prompts, along with a 3D ROI box. Our method leverages a pretrained language-image model to steer the synthesis towards a user-provided text prompt, along with a 3D MLP model initialized on an existing NeRF scene to generate the object and blend it into a specified region in the original scene. We allow local editing by localizing a 3D ROI box in the input scene, and blend the content synthesized inside the ROI with the existing scene using a novel volumetric blending technique. To obtain natural looking and view-consistent results, we leverage existing and new geometric priors and 3D augmentations for improving the visual fidelity of the final result. We test our framework both qualitatively and quantitatively on a variety of real 3D scenes and text prompts, demonstrating realistic multi-view consistent results with much flexibility and diversity compared to the baselines. Finally, we show the applicability of our framework for several 3D editing applications, including adding new objects to a scene, removing/replacing/altering existing objects, and texture conversion.

Related papers

• 3D-SceneDreamer: Text-Driven 3D-Consistent Scene Generation [51.64796781728106]
  We propose a generative refinement network to synthesize new contents with higher quality by exploiting the natural image prior to 2D diffusion model and the global 3D information of the current scene. Our approach supports wide variety of scene generation and arbitrary camera trajectories with improved visual quality and 3D consistency.
  arXiv  Detail & Related papers  (2024-03-14T14:31:22Z)
• Disentangled 3D Scene Generation with Layout Learning [109.03233745767062]
  We introduce a method to generate 3D scenes that are disentangled into their component objects. Our key insight is that objects can be discovered by finding parts of a 3D scene that, when rearranged spatially, still produce valid configurations of the same scene. We show that despite its simplicity, our approach successfully generates 3D scenes into individual objects.
  arXiv  Detail & Related papers  (2024-02-26T18:54:15Z)
• ReplaceAnything3D:Text-Guided 3D Scene Editing with Compositional Neural Radiance Fields [13.425973473159406]
  We introduce ReplaceAnything3D model (RAM3D), a novel text-guided 3D scene editing method. Given multi-view images of a scene, a text prompt describing the object to replace, and a text prompt describing the new object, our Erase-and-Replace approach can effectively swap objects in the scene with newly generated content.
  arXiv  Detail & Related papers  (2024-01-31T15:02:26Z)
• GO-NeRF: Generating Virtual Objects in Neural Radiance Fields [75.13534508391852]
  GO-NeRF is capable of utilizing scene context for high-quality and harmonious 3D object generation within an existing NeRF. Our method employs a compositional rendering formulation that allows the generated 3D objects to be seamlessly composited into the scene.
  arXiv  Detail & Related papers  (2024-01-11T08:58:13Z)
• SceneWiz3D: Towards Text-guided 3D Scene Composition [134.71933134180782]
  Existing approaches either leverage large text-to-image models to optimize a 3D representation or train 3D generators on object-centric datasets. We introduce SceneWiz3D, a novel approach to synthesize high-fidelity 3D scenes from text.
  arXiv  Detail & Related papers  (2023-12-13T18:59:30Z)
• Text2NeRF: Text-Driven 3D Scene Generation with Neural Radiance Fields [29.907615852310204]
  We present Text2NeRF, which is able to generate a wide range of 3D scenes purely from a text prompt. Our method requires no additional training data but only a natural language description of the scene as the input.
  arXiv  Detail & Related papers  (2023-05-19T10:58:04Z)
• CompoNeRF: Text-guided Multi-object Compositional NeRF with Editable 3D Scene Layout [13.364394556439992]
  Text-to-3D form plays a crucial role in creating editable 3D scenes for AR/VR. Recent advances have shown promise in merging neural radiance fields (NeRFs) with pre-trained diffusion models for text-to-3D object generation. We propose a novel framework, dubbed CompoNeRF, by integrating an editable 3D scene layout with object-specific and scene-wide guidance mechanisms. Our framework achieves up to a textbf54% improvement by the multi-view CLIP score metric.
  arXiv  Detail & Related papers  (2023-03-24T07:37:09Z)
• Compositional 3D Scene Generation using Locally Conditioned Diffusion [49.5784841881488]
  We introduce textbflocally conditioned diffusion as an approach to compositional scene diffusion. We demonstrate a score distillation sampling--based text-to-3D synthesis pipeline that enables compositional 3D scene generation at a higher fidelity than relevant baselines.
  arXiv  Detail & Related papers  (2023-03-21T22:37:16Z)
• Decomposing NeRF for Editing via Feature Field Distillation [14.628761232614762]
  editing a scene represented by a NeRF is challenging as the underlying connectionist representations are not object-centric or compositional. In this work, we tackle the problem of semantic scene decomposition of NeRFs to enable query-based local editing. We propose to distill the knowledge of off-the-shelf, self-supervised 2D image feature extractors into a 3D feature field optimized in parallel to the radiance field.
  arXiv  Detail & Related papers  (2022-05-31T07:56:09Z)
• Control-NeRF: Editable Feature Volumes for Scene Rendering and Manipulation [58.16911861917018]
  We present a novel method for performing flexible, 3D-aware image content manipulation while enabling high-quality novel view synthesis. Our model couples learnt scene-specific feature volumes with a scene agnostic neural rendering network. We demonstrate various scene manipulations, including mixing scenes, deforming objects and inserting objects into scenes, while still producing photo-realistic results.
  arXiv  Detail & Related papers  (2022-04-22T17:57:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.