Plasticine3D: 3D Non-Rigid Editing with Text Guidance by Multi-View Embedding Optimization

• URL: http://arxiv.org/abs/2312.10111v2
• Date: Tue, 9 Jul 2024 10:57:28 GMT
• Title: Plasticine3D: 3D Non-Rigid Editing with Text Guidance by Multi-View Embedding Optimization
• Authors: Yige Chen, Teng Hu, Yizhe Tang, Siyuan Chen, Ang Chen, Ran Yi,
• Abstract summary: We propose Plasticine3D, a novel text-guided controlled 3D editing pipeline that can perform 3D non-rigid editing. Our work divides the editing process into a geometry editing stage and a texture editing stage to achieve separate control of structure and appearance. For the purpose of fine-grained control, we propose Embedding-Fusion (EF) to blend the original characteristics with the editing objectives in the embedding space.
• Score: 21.8454418337306
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: With the help of Score Distillation Sampling (SDS) and the rapid development of neural 3D representations, some methods have been proposed to perform 3D editing such as adding additional geometries, or overwriting textures. However, generalized 3D non-rigid editing task, which requires changing both the structure (posture or composition) and appearance (texture) of the original object, remains to be challenging in 3D editing field. In this paper, we propose Plasticine3D, a novel text-guided fine-grained controlled 3D editing pipeline that can perform 3D non-rigid editing with large structure deformations. Our work divides the editing process into a geometry editing stage and a texture editing stage to achieve separate control of structure and appearance. In order to maintain the details of the original object from different viewpoints, we propose a Multi-View-Embedding (MVE) Optimization strategy to ensure that the guidance model learns the features of the original object from various viewpoints. For the purpose of fine-grained control, we propose Embedding-Fusion (EF) to blend the original characteristics with the editing objectives in the embedding space, and control the extent of editing by adjusting the fusion rate. Furthermore, in order to address the issue of gradual loss of details during the generation process under high editing intensity, as well as the problem of insignificant editing effects in some scenarios, we propose Score Projection Sampling (SPS) as a replacement of score distillation sampling, which introduces additional optimization phases for editing target enhancement and original detail maintenance, leading to better editing quality. Extensive experiments demonstrate the effectiveness of our method on 3D non-rigid editing tasks

Related papers

• Drag Your Gaussian: Effective Drag-Based Editing with Score Distillation for 3D Gaussian Splatting [55.14822004410817]
  We introduce DYG, an effective 3D drag-based editing method for 3D Gaussian Splatting. It enables precise control over the extent of editing through the input of 3D masks and pairs of control points. DYG integrates the strengths of the implicit triplane representation to establish the geometric scaffold of the editing results.
  arXiv  Detail & Related papers  (2025-01-30T18:51:54Z)
• DragScene: Interactive 3D Scene Editing with Single-view Drag Instructions [9.31257776760014]
  3D editing has shown remarkable capability in editing scenes based on various instructions. Existing methods struggle with achieving intuitive, localized editing. We introduce DragScene, a framework that integrates drag-style editing with diverse 3D representations.
  arXiv  Detail & Related papers  (2024-12-18T07:02:01Z)
• PrEditor3D: Fast and Precise 3D Shape Editing [100.09112677669376]
  We propose a training-free approach to 3D editing that enables the editing of a single shape within a few minutes. The edited 3D mesh aligns well with the prompts, and remains identical for regions that are not intended to be altered.
  arXiv  Detail & Related papers  (2024-12-09T15:44:47Z)
• ProEdit: Simple Progression is All You Need for High-Quality 3D Scene Editing [33.42456524414643]
  ProEdit is a framework for high-quality 3D scene editing guided by diffusion distillation. Our framework controls the size of FOS and reduces inconsistency by decomposing the overall editing task into several subtasks. ProEdit achieves state-of-the-art results in various scenes and challenging editing tasks.
  arXiv  Detail & Related papers  (2024-11-07T18:59:54Z)
• View-Consistent 3D Editing with Gaussian Splatting [50.6460814430094]
  View-consistent Editing (VcEdit) is a novel framework that seamlessly incorporates 3DGS into image editing processes. By incorporating consistency modules into an iterative pattern, VcEdit proficiently resolves the issue of multi-view inconsistency.
  arXiv  Detail & Related papers  (2024-03-18T15:22:09Z)
• SHAP-EDITOR: Instruction-guided Latent 3D Editing in Seconds [73.91114735118298]
  Shap-Editor is a novel feed-forward 3D editing framework. We demonstrate that direct 3D editing in this space is possible and efficient by building a feed-forward editor network.
  arXiv  Detail & Related papers  (2023-12-14T18:59:06Z)
• Object-aware Inversion and Reassembly for Image Editing [61.19822563737121]
  We propose Object-aware Inversion and Reassembly (OIR) to enable object-level fine-grained editing. We use our search metric to find the optimal inversion step for each editing pair when editing an image. Our method achieves superior performance in editing object shapes, colors, materials, categories, etc., especially in multi-object editing scenarios.
  arXiv  Detail & Related papers  (2023-10-18T17:59:02Z)
• Editing 3D Scenes via Text Prompts without Retraining [80.57814031701744]
  DN2N is a text-driven editing method that allows for the direct acquisition of a NeRF model with universal editing capabilities. Our method employs off-the-shelf text-based editing models of 2D images to modify the 3D scene images. Our method achieves multiple editing types, including but not limited to appearance editing, weather transition, material changing, and style transfer.
  arXiv  Detail & Related papers  (2023-09-10T02:31:50Z)
• SINE: Semantic-driven Image-based NeRF Editing with Prior-guided Editing Field [37.8162035179377]
  We present a novel semantic-driven NeRF editing approach, which enables users to edit a neural radiance field with a single image. To achieve this goal, we propose a prior-guided editing field to encode fine-grained geometric and texture editing in 3D space. Our method achieves photo-realistic 3D editing using only a single edited image, pushing the bound of semantic-driven editing in 3D real-world scenes.
  arXiv  Detail & Related papers  (2023-03-23T13:58:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.