Instruct-NeRF2NeRF: Editing 3D Scenes with Instructions

• URL: http://arxiv.org/abs/2303.12789v2
• Date: Thu, 1 Jun 2023 17:17:38 GMT
• Title: Instruct-NeRF2NeRF: Editing 3D Scenes with Instructions
• Authors: Ayaan Haque, Matthew Tancik, Alexei A. Efros, Aleksander Holynski, Angjoo Kanazawa
• Abstract summary: We propose a method for editing NeRF scenes with text-instructions. Given a NeRF of a scene and the collection of images used to reconstruct it, our method uses an image-conditioned diffusion model (InstructPix2Pix) to iteratively edit the input images while optimizing the underlying scene. We demonstrate that our proposed method is able to edit large-scale, real-world scenes, and is able to accomplish more realistic, targeted edits than prior work.
• Score: 109.51624993088687
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We propose a method for editing NeRF scenes with text-instructions. Given a NeRF of a scene and the collection of images used to reconstruct it, our method uses an image-conditioned diffusion model (InstructPix2Pix) to iteratively edit the input images while optimizing the underlying scene, resulting in an optimized 3D scene that respects the edit instruction. We demonstrate that our proposed method is able to edit large-scale, real-world scenes, and is able to accomplish more realistic, targeted edits than prior work.

Related papers

• NeRF-Insert: 3D Local Editing with Multimodal Control Signals [97.91172669905578]
  NeRF-Insert is a NeRF editing framework that allows users to make high-quality local edits with a flexible level of control. We cast scene editing as an in-painting problem, which encourages the global structure of the scene to be preserved. Our results show better visual quality and also maintain stronger consistency with the original NeRF.
  arXiv  Detail & Related papers  (2024-04-30T02:04:49Z)
• DATENeRF: Depth-Aware Text-based Editing of NeRFs [49.08848777124736]
  We introduce an inpainting approach that leverages the depth information of NeRF scenes to distribute 2D edits across different images. Our results reveal that this methodology achieves more consistent, lifelike, and detailed edits than existing leading methods for text-driven NeRF scene editing.
  arXiv  Detail & Related papers  (2024-04-06T06:48:16Z)
• InstructBrush: Learning Attention-based Instruction Optimization for Image Editing [54.07526261513434]
  InstructBrush is an inversion method for instruction-based image editing methods. It extracts editing effects from image pairs as editing instructions, which are further applied for image editing. Our approach achieves superior performance in editing and is more semantically consistent with the target editing effects.
  arXiv  Detail & Related papers  (2024-03-27T15:03:38Z)
• Customize your NeRF: Adaptive Source Driven 3D Scene Editing via Local-Global Iterative Training [61.984277261016146]
  We propose a CustomNeRF model that unifies a text description or a reference image as the editing prompt. To tackle the first challenge, we propose a Local-Global Iterative Editing (LGIE) training scheme that alternates between foreground region editing and full-image editing. For the second challenge, we also design a class-guided regularization that exploits class priors within the generation model to alleviate the inconsistency problem.
  arXiv  Detail & Related papers  (2023-12-04T06:25:06Z)
• Editing 3D Scenes via Text Prompts without Retraining [80.57814031701744]
  DN2N is a text-driven editing method that allows for the direct acquisition of a NeRF model with universal editing capabilities. Our method employs off-the-shelf text-based editing models of 2D images to modify the 3D scene images. Our method achieves multiple editing types, including but not limited to appearance editing, weather transition, material changing, and style transfer.
  arXiv  Detail & Related papers  (2023-09-10T02:31:50Z)
• Seal-3D: Interactive Pixel-Level Editing for Neural Radiance Fields [14.803266838721864]
  Seal-3D allows users to edit NeRF models in a pixel-level and free manner with a wide range of NeRF-like backbone and preview the editing effects instantly. A NeRF editing system is built to showcase various editing types.
  arXiv  Detail & Related papers  (2023-07-27T18:08:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.