LatentEditor: Text Driven Local Editing of 3D Scenes

• URL: http://arxiv.org/abs/2312.09313v4
• Date: Sat, 13 Jul 2024 19:04:43 GMT
• Title: LatentEditor: Text Driven Local Editing of 3D Scenes
• Authors: Umar Khalid, Hasan Iqbal, Nazmul Karim, Jing Hua, Chen Chen,
• Abstract summary: We introduce textscLatentEditor, a framework for precise and locally controlled editing of neural fields using text prompts. We successfully embed real-world scenes into the latent space, resulting in a faster and more adaptable NeRF backbone for editing. Our approach achieves faster editing speeds and superior output quality compared to existing 3D editing models.
• Score: 8.966537479017951
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While neural fields have made significant strides in view synthesis and scene reconstruction, editing them poses a formidable challenge due to their implicit encoding of geometry and texture information from multi-view inputs. In this paper, we introduce \textsc{LatentEditor}, an innovative framework designed to empower users with the ability to perform precise and locally controlled editing of neural fields using text prompts. Leveraging denoising diffusion models, we successfully embed real-world scenes into the latent space, resulting in a faster and more adaptable NeRF backbone for editing compared to traditional methods. To enhance editing precision, we introduce a delta score to calculate the 2D mask in the latent space that serves as a guide for local modifications while preserving irrelevant regions. Our novel pixel-level scoring approach harnesses the power of InstructPix2Pix (IP2P) to discern the disparity between IP2P conditional and unconditional noise predictions in the latent space. The edited latents conditioned on the 2D masks are then iteratively updated in the training set to achieve 3D local editing. Our approach achieves faster editing speeds and superior output quality compared to existing 3D editing models, bridging the gap between textual instructions and high-quality 3D scene editing in latent space. We show the superiority of our approach on four benchmark 3D datasets, LLFF, IN2N, NeRFStudio and NeRF-Art. Project Page: https://latenteditor.github.io/

Related papers

• GSEditPro: 3D Gaussian Splatting Editing with Attention-based Progressive Localization [11.170354299559998]
  We propose GSEditPro, a novel 3D scene editing framework which allows users to perform various creative and precise editing using text prompts only. We introduce an attention-based progressive localization module to add semantic labels to each Gaussian during rendering. This enables precise localization on editing areas by classifying Gaussians based on their relevance to the editing prompts derived from cross-attention layers of the T2I model.
  arXiv  Detail & Related papers  (2024-11-15T08:25:14Z)
• DragGaussian: Enabling Drag-style Manipulation on 3D Gaussian Representation [57.406031264184584]
  DragGaussian is a 3D object drag-editing framework based on 3D Gaussian Splatting. Our contributions include the introduction of a new task, the development of DragGaussian for interactive point-based 3D editing, and comprehensive validation of its effectiveness through qualitative and quantitative experiments.
  arXiv  Detail & Related papers  (2024-05-09T14:34:05Z)
• ViCA-NeRF: View-Consistency-Aware 3D Editing of Neural Radiance Fields [45.020585071312475]
  ViCA-NeRF is the first view-consistency-aware method for 3D editing with text instructions. We exploit two sources of regularization that explicitly propagate the editing information across different views.
  arXiv  Detail & Related papers  (2024-02-01T18:59:09Z)
• SHAP-EDITOR: Instruction-guided Latent 3D Editing in Seconds [73.91114735118298]
  Shap-Editor is a novel feed-forward 3D editing framework. We demonstrate that direct 3D editing in this space is possible and efficient by building a feed-forward editor network.
  arXiv  Detail & Related papers  (2023-12-14T18:59:06Z)
• MaTe3D: Mask-guided Text-based 3D-aware Portrait Editing [61.014328598895524]
  We propose textbfMaTe3D: mask-guided text-based 3D-aware portrait editing. New SDF-based 3D generator learns local and global representations with proposed SDF and density consistency losses. Conditional Distillation on Geometry and Texture (CDGT) mitigates visual ambiguity and avoids mismatch between texture and geometry.
  arXiv  Detail & Related papers  (2023-12-12T03:04:08Z)
• Learning Naturally Aggregated Appearance for Efficient 3D Editing [94.47518916521065]
  We propose to replace the color field with an explicit 2D appearance aggregation, also called canonical image. To avoid the distortion effect and facilitate convenient editing, we complement the canonical image with a projection field that maps 3D points onto 2D pixels for texture lookup. Our representation, dubbed AGAP, well supports various ways of 3D editing (e.g., stylization, interactive drawing, and content extraction) with no need of re-optimization.
  arXiv  Detail & Related papers  (2023-12-11T18:59:31Z)
• BluNF: Blueprint Neural Field [10.110885977110447]
  We introduce a novel approach, called Blueprint Neural Field (BluNF), to address these editing issues. BluNF provides a robust and user-friendly 2D blueprint, enabling intuitive scene editing. We demonstrate BluNF's editability through an intuitive click-and-change mechanism, enabling 3D manipulations.
  arXiv  Detail & Related papers  (2023-09-07T17:53:25Z)
• DreamEditor: Text-Driven 3D Scene Editing with Neural Fields [115.07896366760876]
  We propose a novel framework that enables users to edit neural fields using text prompts. DreamEditor generates highly realistic textures and geometry, significantly surpassing previous works in both quantitative and qualitative evaluations.
  arXiv  Detail & Related papers  (2023-06-23T11:53:43Z)
• SKED: Sketch-guided Text-based 3D Editing [49.019881133348775]
  We present SKED, a technique for editing 3D shapes represented by NeRFs. Our technique utilizes as few as two guiding sketches from different views to alter an existing neural field. We propose novel loss functions to generate the desired edits while preserving the density and radiance of the base instance.
  arXiv  Detail & Related papers  (2023-03-19T18:40:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.