ProteusNeRF: Fast Lightweight NeRF Editing using 3D-Aware Image Context

• URL: http://arxiv.org/abs/2310.09965v3
• Date: Tue, 23 Apr 2024 10:03:59 GMT
• Title: ProteusNeRF: Fast Lightweight NeRF Editing using 3D-Aware Image Context
• Authors: Binglun Wang, Niladri Shekhar Dutt, Niloy J. Mitra,
• Abstract summary: We present a very simple but effective neural network architecture that is fast and efficient while maintaining a low memory footprint. Our representation allows straightforward object selection via semantic feature distillation at the training stage. We propose a local 3D-aware image context to facilitate view-consistent image editing that can then be distilled into fine-tuned NeRFs.
• Score: 26.07841568311428
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Neural Radiance Fields (NeRFs) have recently emerged as a popular option for photo-realistic object capture due to their ability to faithfully capture high-fidelity volumetric content even from handheld video input. Although much research has been devoted to efficient optimization leading to real-time training and rendering, options for interactive editing NeRFs remain limited. We present a very simple but effective neural network architecture that is fast and efficient while maintaining a low memory footprint. This architecture can be incrementally guided through user-friendly image-based edits. Our representation allows straightforward object selection via semantic feature distillation at the training stage. More importantly, we propose a local 3D-aware image context to facilitate view-consistent image editing that can then be distilled into fine-tuned NeRFs, via geometric and appearance adjustments. We evaluate our setup on a variety of examples to demonstrate appearance and geometric edits and report 10-30x speedup over concurrent work focusing on text-guided NeRF editing. Video results can be seen on our project webpage at https://proteusnerf.github.io.

Related papers

• Taming Rectified Flow for Inversion and Editing [57.3742655030493]
  Rectified-flow-based diffusion transformers, such as FLUX and OpenSora, have demonstrated exceptional performance in the field of image and video generation. Despite their robust generative capabilities, these models often suffer from inaccurate inversion, which could limit their effectiveness in downstream tasks such as image and video editing. We propose RF-r, a novel training-free sampler that enhances inversion precision by reducing errors in the process of solving rectified flow ODEs.
  arXiv  Detail & Related papers  (2024-11-07T14:29:02Z)
• NeRF-Insert: 3D Local Editing with Multimodal Control Signals [97.91172669905578]
  NeRF-Insert is a NeRF editing framework that allows users to make high-quality local edits with a flexible level of control. We cast scene editing as an in-painting problem, which encourages the global structure of the scene to be preserved. Our results show better visual quality and also maintain stronger consistency with the original NeRF.
  arXiv  Detail & Related papers  (2024-04-30T02:04:49Z)
• SealD-NeRF: Interactive Pixel-Level Editing for Dynamic Scenes by Neural Radiance Fields [7.678022563694719]
  SealD-NeRF is an extension of Seal-3D for pixel-level editing in dynamic settings. It allows for consistent edits across sequences by mapping editing actions to a specific timeframe.
  arXiv  Detail & Related papers  (2024-02-21T03:45:18Z)
• ED-NeRF: Efficient Text-Guided Editing of 3D Scene with Latent Space NeRF [60.47731445033151]
  We present a novel 3D NeRF editing approach dubbed ED-NeRF. We embed real-world scenes into the latent space of the latent diffusion model (LDM) through a unique refinement layer. This approach enables us to obtain a NeRF backbone that is not only faster but also more amenable to editing.
  arXiv  Detail & Related papers  (2023-10-04T10:28:38Z)
• Seal-3D: Interactive Pixel-Level Editing for Neural Radiance Fields [14.803266838721864]
  Seal-3D allows users to edit NeRF models in a pixel-level and free manner with a wide range of NeRF-like backbone and preview the editing effects instantly. A NeRF editing system is built to showcase various editing types.
  arXiv  Detail & Related papers  (2023-07-27T18:08:19Z)
• RePaint-NeRF: NeRF Editting via Semantic Masks and Diffusion Models [36.236190350126826]
  We propose a novel framework that can take RGB images as input and alter the 3D content in neural scenes. Specifically, we semantically select the target object and a pre-trained diffusion model will guide the NeRF model to generate new 3D objects. Experiment results show that our algorithm is effective for editing 3D objects in NeRF under different text prompts.
  arXiv  Detail & Related papers  (2023-06-09T04:49:31Z)
• FaceDNeRF: Semantics-Driven Face Reconstruction, Prompt Editing and Relighting with Diffusion Models [67.17713009917095]
  We propose Face Diffusion NeRF (FaceDNeRF), a new generative method to reconstruct high-quality Face NeRFs from single images. With carefully designed illumination and identity preserving loss, FaceDNeRF offers users unparalleled control over the editing process.
  arXiv  Detail & Related papers  (2023-06-01T15:14:39Z)
• SINE: Semantic-driven Image-based NeRF Editing with Prior-guided Editing Field [37.8162035179377]
  We present a novel semantic-driven NeRF editing approach, which enables users to edit a neural radiance field with a single image. To achieve this goal, we propose a prior-guided editing field to encode fine-grained geometric and texture editing in 3D space. Our method achieves photo-realistic 3D editing using only a single edited image, pushing the bound of semantic-driven editing in 3D real-world scenes.
  arXiv  Detail & Related papers  (2023-03-23T13:58:11Z)
• BARF: Bundle-Adjusting Neural Radiance Fields [104.97810696435766]
  We propose Bundle-Adjusting Neural Radiance Fields (BARF) for training NeRF from imperfect camera poses. BARF can effectively optimize the neural scene representations and resolve large camera pose misalignment at the same time. This enables view synthesis and localization of video sequences from unknown camera poses, opening up new avenues for visual localization systems.
  arXiv  Detail & Related papers  (2021-04-13T17:59:51Z)
• MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo [52.329580781898116]
  We present MVSNeRF, a novel neural rendering approach that can efficiently reconstruct neural radiance fields for view synthesis. Unlike prior works on neural radiance fields that consider per-scene optimization on densely captured images, we propose a generic deep neural network that can reconstruct radiance fields from only three nearby input views via fast network inference.
  arXiv  Detail & Related papers  (2021-03-29T13:15:23Z)
• pixelNeRF: Neural Radiance Fields from One or Few Images [20.607712035278315]
  pixelNeRF is a learning framework that predicts a continuous neural scene representation conditioned on one or few input images. We conduct experiments on ShapeNet benchmarks for single image novel view synthesis tasks with held-out objects. In all cases, pixelNeRF outperforms current state-of-the-art baselines for novel view synthesis and single image 3D reconstruction.
  arXiv  Detail & Related papers  (2020-12-03T18:59:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.