Creative Birds: Self-Supervised Single-View 3D Style Transfer

• URL: http://arxiv.org/abs/2307.14127v2
• Date: Thu, 27 Jul 2023 04:21:52 GMT
• Title: Creative Birds: Self-Supervised Single-View 3D Style Transfer
• Authors: Renke Wang, Guimin Que, Shuo Chen, Xiang Li, Jun Li, Jian Yang
• Abstract summary: We propose a novel method for single-view 3D style transfer that generates a unique 3D object with both shape and texture transfer. Our focus lies primarily on birds, a popular subject in 3D reconstruction, for which no existing single-view 3D transfer methods have been developed.
• Score: 23.64817899864608
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this paper, we propose a novel method for single-view 3D style transfer that generates a unique 3D object with both shape and texture transfer. Our focus lies primarily on birds, a popular subject in 3D reconstruction, for which no existing single-view 3D transfer methods have been developed.The method we propose seeks to generate a 3D mesh shape and texture of a bird from two single-view images. To achieve this, we introduce a novel shape transfer generator that comprises a dual residual gated network (DRGNet), and a multi-layer perceptron (MLP). DRGNet extracts the features of source and target images using a shared coordinate gate unit, while the MLP generates spatial coordinates for building a 3D mesh. We also introduce a semantic UV texture transfer module that implements textural style transfer using semantic UV segmentation, which ensures consistency in the semantic meaning of the transferred regions. This module can be widely adapted to many existing approaches. Finally, our method constructs a novel 3D bird using a differentiable renderer. Experimental results on the CUB dataset verify that our method achieves state-of-the-art performance on the single-view 3D style transfer task. Code is available in https://github.com/wrk226/creative_birds.

Related papers

• GaussianAnything: Interactive Point Cloud Flow Matching For 3D Object Generation [75.39457097832113]
  This paper introduces a novel 3D generation framework, offering scalable, high-quality 3D generation with an interactive Point Cloud-structured Latent space. Our framework employs a Variational Autoencoder with multi-view posed RGB-D(epth)-N(ormal) renderings as input, using a unique latent space design that preserves 3D shape information. The proposed method, GaussianAnything, supports multi-modal conditional 3D generation, allowing for point cloud, caption, and single image inputs.
  arXiv  Detail & Related papers  (2024-11-12T18:59:32Z)
• Free3D: Consistent Novel View Synthesis without 3D Representation [63.931920010054064]
  Free3D is a simple accurate method for monocular open-set novel view synthesis (NVS) Compared to other works that took a similar approach, we obtain significant improvements without resorting to an explicit 3D representation.
  arXiv  Detail & Related papers  (2023-12-07T18:59:18Z)
• Consistent Mesh Diffusion [8.318075237885857]
  Given a 3D mesh with a UV parameterization, we introduce a novel approach to generating textures from text prompts. We demonstrate our approach on a dataset containing 30 meshes, taking approximately 5 minutes per mesh.
  arXiv  Detail & Related papers  (2023-12-01T23:25:14Z)
• DI-Net : Decomposed Implicit Garment Transfer Network for Digital Clothed 3D Human [75.45488434002898]
  Existing 2D virtual try-on methods cannot be directly extended to 3D since they lack the ability to perceive the depth of each pixel. We propose a Decomposed Implicit garment transfer network (DI-Net), which can effortlessly reconstruct a 3D human mesh with the newly try-on result.
  arXiv  Detail & Related papers  (2023-11-28T14:28:41Z)
• 3DStyle-Diffusion: Pursuing Fine-grained Text-driven 3D Stylization with 2D Diffusion Models [102.75875255071246]
  3D content creation via text-driven stylization has played a fundamental challenge to multimedia and graphics community. We propose a new 3DStyle-Diffusion model that triggers fine-grained stylization of 3D meshes with additional controllable appearance and geometric guidance from 2D Diffusion models.
  arXiv  Detail & Related papers  (2023-11-09T15:51:27Z)
• Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Models [21.622420436349245]
  We present Text2Room, a method for generating room-scale textured 3D meshes from a given text prompt as input. We leverage pre-trained 2D text-to-image models to synthesize a sequence of images from different poses. In order to lift these outputs into a consistent 3D scene representation, we combine monocular depth estimation with a text-conditioned inpainting model.
  arXiv  Detail & Related papers  (2023-03-21T16:21:02Z)
• Self-Supervised Geometry-Aware Encoder for Style-Based 3D GAN Inversion [115.82306502822412]
  StyleGAN has achieved great progress in 2D face reconstruction and semantic editing via image inversion and latent editing. A corresponding generic 3D GAN inversion framework is still missing, limiting the applications of 3D face reconstruction and semantic editing. We study the challenging problem of 3D GAN inversion where a latent code is predicted given a single face image to faithfully recover its 3D shapes and detailed textures.
  arXiv  Detail & Related papers  (2022-12-14T18:49:50Z)
• 3DStyleNet: Creating 3D Shapes with Geometric and Texture Style Variations [81.45521258652734]
  We propose a method to create plausible geometric and texture style variations of 3D objects. Our method can create many novel stylized shapes, resulting in effortless 3D content creation and style-ware data augmentation.
  arXiv  Detail & Related papers  (2021-08-30T02:28:31Z)
• 3DSNet: Unsupervised Shape-to-Shape 3D Style Transfer [66.48720190245616]
  We propose a learning-based approach for style transfer between 3D objects. The proposed method can synthesize new 3D shapes both in the form of point clouds and meshes. We extend our technique to implicitly learn the multimodal style distribution of the chosen domains.
  arXiv  Detail & Related papers  (2020-11-26T16:59:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.