GenN2N: Generative NeRF2NeRF Translation

• URL: http://arxiv.org/abs/2404.02788v1
• Date: Wed, 3 Apr 2024 14:56:06 GMT
• Title: GenN2N: Generative NeRF2NeRF Translation
• Authors: Xiangyue Liu, Han Xue, Kunming Luo, Ping Tan, Li Yi,
• Abstract summary: GenN2N is a unified NeRF-to-NeRF translation framework for various NeRF translation tasks. It employs a plug-and-play image-to-image translator to perform editing in the 2D domain and lifting 2D edits into the 3D NeRF space.
• Score: 53.20986183316661
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present GenN2N, a unified NeRF-to-NeRF translation framework for various NeRF translation tasks such as text-driven NeRF editing, colorization, super-resolution, inpainting, etc. Unlike previous methods designed for individual translation tasks with task-specific schemes, GenN2N achieves all these NeRF editing tasks by employing a plug-and-play image-to-image translator to perform editing in the 2D domain and lifting 2D edits into the 3D NeRF space. Since the 3D consistency of 2D edits may not be assured, we propose to model the distribution of the underlying 3D edits through a generative model that can cover all possible edited NeRFs. To model the distribution of 3D edited NeRFs from 2D edited images, we carefully design a VAE-GAN that encodes images while decoding NeRFs. The latent space is trained to align with a Gaussian distribution and the NeRFs are supervised through an adversarial loss on its renderings. To ensure the latent code does not depend on 2D viewpoints but truly reflects the 3D edits, we also regularize the latent code through a contrastive learning scheme. Extensive experiments on various editing tasks show GenN2N, as a universal framework, performs as well or better than task-specific specialists while possessing flexible generative power. More results on our project page: https://xiangyueliu.github.io/GenN2N/

Related papers

• DragGaussian: Enabling Drag-style Manipulation on 3D Gaussian Representation [57.406031264184584]
  DragGaussian is a 3D object drag-editing framework based on 3D Gaussian Splatting. Our contributions include the introduction of a new task, the development of DragGaussian for interactive point-based 3D editing, and comprehensive validation of its effectiveness through qualitative and quantitative experiments.
  arXiv  Detail & Related papers  (2024-05-09T14:34:05Z)
• ViCA-NeRF: View-Consistency-Aware 3D Editing of Neural Radiance Fields [45.020585071312475]
  ViCA-NeRF is the first view-consistency-aware method for 3D editing with text instructions. We exploit two sources of regularization that explicitly propagate the editing information across different views.
  arXiv  Detail & Related papers  (2024-02-01T18:59:09Z)
• LatentEditor: Text Driven Local Editing of 3D Scenes [8.966537479017951]
  We introduce textscLatentEditor, a framework for precise and locally controlled editing of neural fields using text prompts. We successfully embed real-world scenes into the latent space, resulting in a faster and more adaptable NeRF backbone for editing. Our approach achieves faster editing speeds and superior output quality compared to existing 3D editing models.
  arXiv  Detail & Related papers  (2023-12-14T19:38:06Z)
• ED-NeRF: Efficient Text-Guided Editing of 3D Scene with Latent Space NeRF [60.47731445033151]
  We present a novel 3D NeRF editing approach dubbed ED-NeRF. We embed real-world scenes into the latent space of the latent diffusion model (LDM) through a unique refinement layer. This approach enables us to obtain a NeRF backbone that is not only faster but also more amenable to editing.
  arXiv  Detail & Related papers  (2023-10-04T10:28:38Z)
• RePaint-NeRF: NeRF Editting via Semantic Masks and Diffusion Models [36.236190350126826]
  We propose a novel framework that can take RGB images as input and alter the 3D content in neural scenes. Specifically, we semantically select the target object and a pre-trained diffusion model will guide the NeRF model to generate new 3D objects. Experiment results show that our algorithm is effective for editing 3D objects in NeRF under different text prompts.
  arXiv  Detail & Related papers  (2023-06-09T04:49:31Z)
• FaceDNeRF: Semantics-Driven Face Reconstruction, Prompt Editing and Relighting with Diffusion Models [67.17713009917095]
  We propose Face Diffusion NeRF (FaceDNeRF), a new generative method to reconstruct high-quality Face NeRFs from single images. With carefully designed illumination and identity preserving loss, FaceDNeRF offers users unparalleled control over the editing process.
  arXiv  Detail & Related papers  (2023-06-01T15:14:39Z)
• FeatureNeRF: Learning Generalizable NeRFs by Distilling Foundation Models [21.523836478458524]
  Recent works on generalizable NeRFs have shown promising results on novel view synthesis from single or few images. We propose a novel framework named FeatureNeRF to learn generalizable NeRFs by distilling pre-trained vision models. Our experiments demonstrate the effectiveness of FeatureNeRF as a generalizable 3D semantic feature extractor.
  arXiv  Detail & Related papers  (2023-03-22T17:57:01Z)
• Removing Objects From Neural Radiance Fields [60.067117643543824]
  We propose a framework to remove objects from a NeRF representation created from an RGB-D sequence. Our NeRF inpainting method leverages recent work in 2D image inpainting and is guided by a user-provided mask. We show that our method for NeRF editing is effective for synthesizing plausible inpaintings in a multi-view coherent manner.
  arXiv  Detail & Related papers  (2022-12-22T18:51:06Z)
• Sem2NeRF: Converting Single-View Semantic Masks to Neural Radiance Fields [49.41982694533966]
  We introduce a new task, Semantic-to-NeRF translation, conditioned on one single-view semantic mask as input. In particular, Sem2NeRF addresses the highly challenging task by encoding the semantic mask into the latent code that controls the 3D scene representation of a pretrained decoder. We verify the efficacy of the proposed Sem2NeRF and demonstrate it outperforms several strong baselines on two benchmark datasets.
  arXiv  Detail & Related papers  (2022-03-21T09:15:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.