ShapeFusion: A 3D diffusion model for localized shape editing

• URL: http://arxiv.org/abs/2403.19773v2
• Date: Thu, 4 Apr 2024 09:05:49 GMT
• Title: ShapeFusion: A 3D diffusion model for localized shape editing
• Authors: Rolandos Alexandros Potamias, Michail Tarasiou, Stylianos Ploumpis, Stefanos Zafeiriou,
• Abstract summary: We propose an effective diffusion masking training strategy that, by design, facilitates localized manipulation of any shape region. Compared to the current state-of-the-art our method leads to more interpretable shape manipulations than methods relying on latent code state.
• Score: 37.82690898932135
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In the realm of 3D computer vision, parametric models have emerged as a ground-breaking methodology for the creation of realistic and expressive 3D avatars. Traditionally, they rely on Principal Component Analysis (PCA), given its ability to decompose data to an orthonormal space that maximally captures shape variations. However, due to the orthogonality constraints and the global nature of PCA's decomposition, these models struggle to perform localized and disentangled editing of 3D shapes, which severely affects their use in applications requiring fine control such as face sculpting. In this paper, we leverage diffusion models to enable diverse and fully localized edits on 3D meshes, while completely preserving the un-edited regions. We propose an effective diffusion masking training strategy that, by design, facilitates localized manipulation of any shape region, without being limited to predefined regions or to sparse sets of predefined control vertices. Following our framework, a user can explicitly set their manipulation region of choice and define an arbitrary set of vertices as handles to edit a 3D mesh. Compared to the current state-of-the-art our method leads to more interpretable shape manipulations than methods relying on latent code state, greater localization and generation diversity while offering faster inference than optimization based approaches. Project page: https://rolpotamias.github.io/Shapefusion/

Related papers

• 3D Gaussian Editing with A Single Image [19.662680524312027]
  We introduce a novel single-image-driven 3D scene editing approach based on 3D Gaussian Splatting. Our method learns to optimize the 3D Gaussians to align with an edited version of the image rendered from a user-specified viewpoint. Experiments show the effectiveness of our method in handling geometric details, long-range, and non-rigid deformation.
  arXiv  Detail & Related papers  (2024-08-14T13:17:42Z)
• Locally Adaptive Neural 3D Morphable Models [38.38400553022714]
  We present the Locally Adaptive Morphable Model (LAMM), a framework for learning to generate and manipulate 3D meshes. A very efficient computational graph allows our network to train with only a fraction of the memory required by previous methods. We further leverage local geometry control as a primitive for higher level editing operations and present a set of derivative capabilities.
  arXiv  Detail & Related papers  (2024-01-05T18:28:51Z)
• Learning Naturally Aggregated Appearance for Efficient 3D Editing [94.47518916521065]
  We propose to replace the color field with an explicit 2D appearance aggregation, also called canonical image. To avoid the distortion effect and facilitate convenient editing, we complement the canonical image with a projection field that maps 3D points onto 2D pixels for texture lookup. Our representation, dubbed AGAP, well supports various ways of 3D editing (e.g., stylization, interactive drawing, and content extraction) with no need of re-optimization.
  arXiv  Detail & Related papers  (2023-12-11T18:59:31Z)
• DragD3D: Realistic Mesh Editing with Rigidity Control Driven by 2D Diffusion Priors [10.355568895429588]
  Direct mesh editing and deformation are key components in the geometric modeling and animation pipeline. Regularizers are not aware of the global context and semantics of the object. We show that our deformations can be controlled to yield realistic shape deformations aware of the global context.
  arXiv  Detail & Related papers  (2023-10-06T19:55:40Z)
• DeformToon3D: Deformable 3D Toonification from Neural Radiance Fields [96.0858117473902]
  3D toonification involves transferring the style of an artistic domain onto a target 3D face with stylized geometry and texture. We propose DeformToon3D, an effective toonification framework tailored for hierarchical 3D GAN. Our approach decomposes 3D toonification into subproblems of geometry and texture stylization to better preserve the original latent space.
  arXiv  Detail & Related papers  (2023-09-08T16:17:45Z)
• 3Deformer: A Common Framework for Image-Guided Mesh Deformation [27.732389685912214]
  Given a source 3D mesh with semantic materials, and a user-specified semantic image, 3Deformer can accurately edit the source mesh. Our 3Deformer is able to produce impressive results and reaches the state-of-the-art level.
  arXiv  Detail & Related papers  (2023-07-19T10:44:44Z)
• Differentiable Blocks World: Qualitative 3D Decomposition by Rendering Primitives [70.32817882783608]
  We present an approach that produces a simple, compact, and actionable 3D world representation by means of 3D primitives. Unlike existing primitive decomposition methods that rely on 3D input data, our approach operates directly on images. We show that the resulting textured primitives faithfully reconstruct the input images and accurately model the visible 3D points.
  arXiv  Detail & Related papers  (2023-07-11T17:58:31Z)
• Implicit Neural Head Synthesis via Controllable Local Deformation Fields [12.191729556779972]
  We build on part-based implicit shape models that decompose a global deformation field into local ones. Our novel formulation models multiple implicit deformation fields with local semantic rig-like control via 3DMM-based parameters. Our formulation renders sharper locally controllable nonlinear deformations than previous implicit monocular approaches.
  arXiv  Detail & Related papers  (2023-04-21T16:35:28Z)
• Pop-Out Motion: 3D-Aware Image Deformation via Learning the Shape Laplacian [58.704089101826774]
  We present a 3D-aware image deformation method with minimal restrictions on shape category and deformation type. We take a supervised learning-based approach to predict the shape Laplacian of the underlying volume of a 3D reconstruction represented as a point cloud. In the experiments, we present our results of deforming 2D character and clothed human images.
  arXiv  Detail & Related papers  (2022-03-29T04:57:18Z)
• SNARF: Differentiable Forward Skinning for Animating Non-Rigid Neural Implicit Shapes [117.76767853430243]
  We introduce SNARF, which combines the advantages of linear blend skinning for polygonal meshes with neural implicit surfaces. We propose a forward skinning model that finds all canonical correspondences of any deformed point using iterative root finding. Compared to state-of-the-art neural implicit representations, our approach generalizes better to unseen poses while preserving accuracy.
  arXiv  Detail & Related papers  (2021-04-08T17:54:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.