CharacterGen: Efficient 3D Character Generation from Single Images with Multi-View Pose Canonicalization

• URL: http://arxiv.org/abs/2402.17214v3
• Date: Wed, 10 Jul 2024 14:25:08 GMT
• Title: CharacterGen: Efficient 3D Character Generation from Single Images with Multi-View Pose Canonicalization
• Authors: Hao-Yang Peng, Jia-Peng Zhang, Meng-Hao Guo, Yan-Pei Cao, Shi-Min Hu,
• Abstract summary: We present CharacterGen, a framework developed to efficiently generate 3D characters. A transformer-based, generalizable sparse-view reconstruction model is the other core component of our approach. We have curated a dataset of anime characters, rendered in multiple poses and views, to train and evaluate our model.
• Score: 27.55341255800119
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: In the field of digital content creation, generating high-quality 3D characters from single images is challenging, especially given the complexities of various body poses and the issues of self-occlusion and pose ambiguity. In this paper, we present CharacterGen, a framework developed to efficiently generate 3D characters. CharacterGen introduces a streamlined generation pipeline along with an image-conditioned multi-view diffusion model. This model effectively calibrates input poses to a canonical form while retaining key attributes of the input image, thereby addressing the challenges posed by diverse poses. A transformer-based, generalizable sparse-view reconstruction model is the other core component of our approach, facilitating the creation of detailed 3D models from multi-view images. We also adopt a texture-back-projection strategy to produce high-quality texture maps. Additionally, we have curated a dataset of anime characters, rendered in multiple poses and views, to train and evaluate our model. Our approach has been thoroughly evaluated through quantitative and qualitative experiments, showing its proficiency in generating 3D characters with high-quality shapes and textures, ready for downstream applications such as rigging and animation.

Related papers

• StdGEN: Semantic-Decomposed 3D Character Generation from Single Images [28.302030751098354]
  StdGEN is an innovative pipeline for generating semantically high-quality 3D characters from single images. It generates intricately detailed 3D characters with separated semantic components such as the body, clothes, and hair, in three minutes. StdGEN offers ready-to-use semantic-decomposed 3D characters and enables flexible customization for a wide range of applications.
  arXiv  Detail & Related papers  (2024-11-08T17:54:18Z)
• Flex3D: Feed-Forward 3D Generation With Flexible Reconstruction Model And Input View Curation [61.040832373015014]
  We propose Flex3D, a novel framework for generating high-quality 3D content from text, single images, or sparse view images. We employ a fine-tuned multi-view image diffusion model and a video diffusion model to generate a pool of candidate views, enabling a rich representation of the target 3D object. In the second stage, the curated views are fed into a Flexible Reconstruction Model (FlexRM), built upon a transformer architecture that can effectively process an arbitrary number of inputs.
  arXiv  Detail & Related papers  (2024-10-01T17:29:43Z)
• ID-to-3D: Expressive ID-guided 3D Heads via Score Distillation Sampling [96.87575334960258]
  ID-to-3D is a method to generate identity- and text-guided 3D human heads with disentangled expressions. Results achieve an unprecedented level of identity-consistent and high-quality texture and geometry generation.
  arXiv  Detail & Related papers  (2024-05-26T13:36:45Z)
• Envision3D: One Image to 3D with Anchor Views Interpolation [18.31796952040799]
  We present Envision3D, a novel method for efficiently generating high-quality 3D content from a single image. It is capable of generating high-quality 3D content in terms of texture and geometry, surpassing previous image-to-3D baseline methods.
  arXiv  Detail & Related papers  (2024-03-13T18:46:33Z)
• ViewDiff: 3D-Consistent Image Generation with Text-to-Image Models [65.22994156658918]
  We present a method that learns to generate multi-view images in a single denoising process from real-world data. We design an autoregressive generation that renders more 3D-consistent images at any viewpoint.
  arXiv  Detail & Related papers  (2024-03-04T07:57:05Z)
• En3D: An Enhanced Generative Model for Sculpting 3D Humans from 2D Synthetic Data [36.51674664590734]
  We present En3D, an enhanced izable scheme for high-qualityd 3D human avatars. Unlike previous works that rely on scarce 3D datasets or limited 2D collections with imbalance viewing angles and pose priors, our approach aims to develop a zero-shot 3D capable of producing 3D humans.
  arXiv  Detail & Related papers  (2024-01-02T12:06:31Z)
• Guide3D: Create 3D Avatars from Text and Image Guidance [55.71306021041785]
  Guide3D is a text-and-image-guided generative model for 3D avatar generation based on diffusion models. Our framework produces topologically and structurally correct geometry and high-resolution textures.
  arXiv  Detail & Related papers  (2023-08-18T17:55:47Z)
• High-fidelity 3D GAN Inversion by Pseudo-multi-view Optimization [51.878078860524795]
  We present a high-fidelity 3D generative adversarial network (GAN) inversion framework that can synthesize photo-realistic novel views. Our approach enables high-fidelity 3D rendering from a single image, which is promising for various applications of AI-generated 3D content.
  arXiv  Detail & Related papers  (2022-11-28T18:59:52Z)
• Pixel2Mesh++: 3D Mesh Generation and Refinement from Multi-View Images [82.32776379815712]
  We study the problem of shape generation in 3D mesh representation from a small number of color images with or without camera poses. We adopt to further improve the shape quality by leveraging cross-view information with a graph convolution network. Our model is robust to the quality of the initial mesh and the error of camera pose, and can be combined with a differentiable function for test-time optimization.
  arXiv  Detail & Related papers  (2022-04-21T03:42:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.