Perturb-and-Revise: Flexible 3D Editing with Generative Trajectories

• URL: http://arxiv.org/abs/2412.05279v1
• Date: Fri, 06 Dec 2024 18:59:53 GMT
• Title: Perturb-and-Revise: Flexible 3D Editing with Generative Trajectories
• Authors: Susung Hong, Johanna Karras, Ricardo Martin-Brualla, Ira Kemelmacher-Shlizerman,
• Abstract summary: Existing 3D editing methods struggle with extensive geometric or appearance changes. We propose Perturb-and-Revise, which makes possible a variety of NeRF editing. We show that Perturb-and-Revise facilitates flexible, effective, and consistent editing of color, appearance, and geometry in 3D.
• Score: 14.403738312095362
• License:
• Abstract: The fields of 3D reconstruction and text-based 3D editing have advanced significantly with the evolution of text-based diffusion models. While existing 3D editing methods excel at modifying color, texture, and style, they struggle with extensive geometric or appearance changes, thus limiting their applications. We propose Perturb-and-Revise, which makes possible a variety of NeRF editing. First, we perturb the NeRF parameters with random initializations to create a versatile initialization. We automatically determine the perturbation magnitude through analysis of the local loss landscape. Then, we revise the edited NeRF via generative trajectories. Combined with the generative process, we impose identity-preserving gradients to refine the edited NeRF. Extensive experiments demonstrate that Perturb-and-Revise facilitates flexible, effective, and consistent editing of color, appearance, and geometry in 3D. For 360{\deg} results, please visit our project page: https://susunghong.github.io/Perturb-and-Revise.

Related papers

• Drag Your Gaussian: Effective Drag-Based Editing with Score Distillation for 3D Gaussian Splatting [55.14822004410817]
  We introduce DYG, an effective 3D drag-based editing method for 3D Gaussian Splatting. It enables precise control over the extent of editing through the input of 3D masks and pairs of control points. DYG integrates the strengths of the implicit triplane representation to establish the geometric scaffold of the editing results.
  arXiv  Detail & Related papers  (2025-01-30T18:51:54Z)
• GSEditPro: 3D Gaussian Splatting Editing with Attention-based Progressive Localization [11.170354299559998]
  We propose GSEditPro, a novel 3D scene editing framework which allows users to perform various creative and precise editing using text prompts only. We introduce an attention-based progressive localization module to add semantic labels to each Gaussian during rendering. This enables precise localization on editing areas by classifying Gaussians based on their relevance to the editing prompts derived from cross-attention layers of the T2I model.
  arXiv  Detail & Related papers  (2024-11-15T08:25:14Z)
• DragGaussian: Enabling Drag-style Manipulation on 3D Gaussian Representation [57.406031264184584]
  DragGaussian is a 3D object drag-editing framework based on 3D Gaussian Splatting. Our contributions include the introduction of a new task, the development of DragGaussian for interactive point-based 3D editing, and comprehensive validation of its effectiveness through qualitative and quantitative experiments.
  arXiv  Detail & Related papers  (2024-05-09T14:34:05Z)
• Text-Guided 3D Face Synthesis -- From Generation to Editing [53.86765812392627]
  We propose a unified text-guided framework from face generation to editing. We employ a fine-tuned texture diffusion model to enhance texture quality in both RGB and YUV space. We propose a self-guided consistency weight strategy to improve editing efficacy while preserving consistency.
  arXiv  Detail & Related papers  (2023-12-01T06:36:23Z)
• GaussianEditor: Swift and Controllable 3D Editing with Gaussian Splatting [66.08674785436612]
  3D editing plays a crucial role in many areas such as gaming and virtual reality. Traditional 3D editing methods, which rely on representations like meshes and point clouds, often fall short in realistically depicting complex scenes. Our paper presents GaussianEditor, an innovative and efficient 3D editing algorithm based on Gaussian Splatting (GS), a novel 3D representation.
  arXiv  Detail & Related papers  (2023-11-24T14:46:59Z)
• ED-NeRF: Efficient Text-Guided Editing of 3D Scene with Latent Space NeRF [60.47731445033151]
  We present a novel 3D NeRF editing approach dubbed ED-NeRF. We embed real-world scenes into the latent space of the latent diffusion model (LDM) through a unique refinement layer. This approach enables us to obtain a NeRF backbone that is not only faster but also more amenable to editing.
  arXiv  Detail & Related papers  (2023-10-04T10:28:38Z)
• Seal-3D: Interactive Pixel-Level Editing for Neural Radiance Fields [14.803266838721864]
  Seal-3D allows users to edit NeRF models in a pixel-level and free manner with a wide range of NeRF-like backbone and preview the editing effects instantly. A NeRF editing system is built to showcase various editing types.
  arXiv  Detail & Related papers  (2023-07-27T18:08:19Z)
• RePaint-NeRF: NeRF Editting via Semantic Masks and Diffusion Models [36.236190350126826]
  We propose a novel framework that can take RGB images as input and alter the 3D content in neural scenes. Specifically, we semantically select the target object and a pre-trained diffusion model will guide the NeRF model to generate new 3D objects. Experiment results show that our algorithm is effective for editing 3D objects in NeRF under different text prompts.
  arXiv  Detail & Related papers  (2023-06-09T04:49:31Z)
• SINE: Semantic-driven Image-based NeRF Editing with Prior-guided Editing Field [37.8162035179377]
  We present a novel semantic-driven NeRF editing approach, which enables users to edit a neural radiance field with a single image. To achieve this goal, we propose a prior-guided editing field to encode fine-grained geometric and texture editing in 3D space. Our method achieves photo-realistic 3D editing using only a single edited image, pushing the bound of semantic-driven editing in 3D real-world scenes.
  arXiv  Detail & Related papers  (2023-03-23T13:58:11Z)
• SKED: Sketch-guided Text-based 3D Editing [49.019881133348775]
  We present SKED, a technique for editing 3D shapes represented by NeRFs. Our technique utilizes as few as two guiding sketches from different views to alter an existing neural field. We propose novel loss functions to generate the desired edits while preserving the density and radiance of the base instance.
  arXiv  Detail & Related papers  (2023-03-19T18:40:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.