StylePart: Image-based Shape Part Manipulation

• URL: http://arxiv.org/abs/2111.10520v2
• Date: Tue, 23 Nov 2021 23:46:46 GMT
• Title: StylePart: Image-based Shape Part Manipulation
• Authors: I-Chao Shen, Li-Wen Su, Yu-Ting Wu, Bing-Yu Chen
• Abstract summary: StylePart is a framework that enables direct shape manipulation of an image by leveraging generative models of both images and 3D shapes. Our key contribution is a shape-consistent latent mapping function that connects the image generative latent space and the 3D man-made shape attribute latent space. We demonstrate our approach through various manipulation tasks, including part replacement, part resizing, and viewpoint manipulation.
• Score: 12.441476696381814
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Due to a lack of image-based "part controllers", shape manipulation of man-made shape images, such as resizing the backrest of a chair or replacing a cup handle is not intuitive. To tackle this problem, we present StylePart, a framework that enables direct shape manipulation of an image by leveraging generative models of both images and 3D shapes. Our key contribution is a shape-consistent latent mapping function that connects the image generative latent space and the 3D man-made shape attribute latent space. Our method "forwardly maps" the image content to its corresponding 3D shape attributes, where the shape part can be easily manipulated. The attribute codes of the manipulated 3D shape are then "backwardly mapped" to the image latent code to obtain the final manipulated image. We demonstrate our approach through various manipulation tasks, including part replacement, part resizing, and viewpoint manipulation, and evaluate its effectiveness through extensive ablation studies.

Related papers

• Michelangelo: Conditional 3D Shape Generation based on Shape-Image-Text Aligned Latent Representation [47.945556996219295]
  We present a novel alignment-before-generation approach to generate 3D shapes based on 2D images or texts. Our framework comprises two models: a Shape-Image-Text-Aligned Variational Auto-Encoder (SITA-VAE) and a conditional Aligned Shape Latent Diffusion Model (ASLDM)
  arXiv  Detail & Related papers  (2023-06-29T17:17:57Z)
• Cross-Modal 3D Shape Generation and Manipulation [62.50628361920725]
  We propose a generic multi-modal generative model that couples the 2D modalities and implicit 3D representations through shared latent spaces. We evaluate our framework on two representative 2D modalities of grayscale line sketches and rendered color images.
  arXiv  Detail & Related papers  (2022-07-24T19:22:57Z)
• Beyond 3DMM: Learning to Capture High-fidelity 3D Face Shape [77.95154911528365]
  3D Morphable Model (3DMM) fitting has widely benefited face analysis due to its strong 3D priori. Previous reconstructed 3D faces suffer from degraded visual verisimilitude due to the loss of fine-grained geometry. This paper proposes a complete solution to capture the personalized shape so that the reconstructed shape looks identical to the corresponding person.
  arXiv  Detail & Related papers  (2022-04-09T03:46:18Z)
• Pop-Out Motion: 3D-Aware Image Deformation via Learning the Shape Laplacian [58.704089101826774]
  We present a 3D-aware image deformation method with minimal restrictions on shape category and deformation type. We take a supervised learning-based approach to predict the shape Laplacian of the underlying volume of a 3D reconstruction represented as a point cloud. In the experiments, we present our results of deforming 2D character and clothed human images.
  arXiv  Detail & Related papers  (2022-03-29T04:57:18Z)
• 3D Shape Reconstruction from 2D Images with Disentangled Attribute Flow [61.62796058294777]
  Reconstructing 3D shape from a single 2D image is a challenging task. Most of the previous methods still struggle to extract semantic attributes for 3D reconstruction task. We propose 3DAttriFlow to disentangle and extract semantic attributes through different semantic levels in the input images.
  arXiv  Detail & Related papers  (2022-03-29T02:03:31Z)
• Learning Canonical 3D Object Representation for Fine-Grained Recognition [77.33501114409036]
  We propose a novel framework for fine-grained object recognition that learns to recover object variation in 3D space from a single image. We represent an object as a composition of 3D shape and its appearance, while eliminating the effect of camera viewpoint. By incorporating 3D shape and appearance jointly in a deep representation, our method learns the discriminative representation of the object.
  arXiv  Detail & Related papers  (2021-08-10T12:19:34Z)
• Discovering 3D Parts from Image Collections [98.16987919686709]
  We tackle the problem of 3D part discovery from only 2D image collections. Instead of relying on manually annotated parts for supervision, we propose a self-supervised approach. Our key insight is to learn a novel part shape prior that allows each part to fit an object shape faithfully while constrained to have simple geometry.
  arXiv  Detail & Related papers  (2021-07-28T20:29:16Z)
• Single-View 3D Object Reconstruction from Shape Priors in Memory [15.641803721287628]
  Existing methods for single-view 3D object reconstruction do not contain enough information to reconstruct high-quality 3D shapes. We propose a novel method, named Mem3D, that explicitly constructs shape priors to supplement the missing information in the image. We also propose a voxel triplet loss function that helps to retrieve the precise 3D shapes that are highly related to the input image from shape priors.
  arXiv  Detail & Related papers  (2020-03-08T03:51:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.