Towards Native Generative Model for 3D Head Avatar

• URL: http://arxiv.org/abs/2410.01226v1
• Date: Wed, 2 Oct 2024 04:04:10 GMT
• Title: Towards Native Generative Model for 3D Head Avatar
• Authors: Yiyu Zhuang, Yuxiao He, Jiawei Zhang, Yanwen Wang, Jiahe Zhu, Yao Yao, Siyu Zhu, Xun Cao, Hao Zhu,
• Abstract summary: We show how to learn a native generative model for 360$circ$ full head from a limited 3D head dataset. Specifically, three major problems are studied: how to effectively utilize various representations for generating the 360$circ$-renderable human head. We hope the proposed models and artist-designed dataset can inspire future research on learning native generative 3D head models from limited 3D datasets.
• Score: 20.770534728078623
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Creating 3D head avatars is a significant yet challenging task for many applicated scenarios. Previous studies have set out to learn 3D human head generative models using massive 2D image data. Although these models are highly generalizable for human appearance, their result models are not 360$^\circ$-renderable, and the predicted 3D geometry is unreliable. Therefore, such results cannot be used in VR, game modeling, and other scenarios that require 360$^\circ$-renderable 3D head models. An intuitive idea is that 3D head models with limited amount but high 3D accuracy are more reliable training data for a high-quality 3D generative model. In this vein, we delve into how to learn a native generative model for 360$^\circ$ full head from a limited 3D head dataset. Specifically, three major problems are studied: 1) how to effectively utilize various representations for generating the 360$^\circ$-renderable human head; 2) how to disentangle the appearance, shape, and motion of human faces to generate a 3D head model that can be edited by appearance and driven by motion; 3) and how to extend the generalization capability of the generative model to support downstream tasks. Comprehensive experiments are conducted to verify the effectiveness of the proposed model. We hope the proposed models and artist-designed dataset can inspire future research on learning native generative 3D head models from limited 3D datasets.

Related papers

• HeadCraft: Modeling High-Detail Shape Variations for Animated 3DMMs [9.790185628415301]
  We introduce a generative model for detailed 3D head meshes on top of an articulated 3DMM. We train a StyleGAN model in order to generalize over the UV maps of displacements. We demonstrate the results of unconditional generation and fitting to the full or partial observation.
  arXiv  Detail & Related papers  (2023-12-21T18:57:52Z)
• GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Models [102.22388340738536]
  2D and 3D diffusion models can generate decent 3D objects based on prompts. 3D diffusion models have good 3D consistency, but their quality and generalization are limited as trainable 3D data is expensive and hard to obtain. This paper attempts to bridge the power from the two types of diffusion models via the recent explicit and efficient 3D Gaussian splatting representation.
  arXiv  Detail & Related papers  (2023-10-12T17:22:24Z)
• Articulated 3D Head Avatar Generation using Text-to-Image Diffusion Models [107.84324544272481]
  The ability to generate diverse 3D articulated head avatars is vital to a plethora of applications, including augmented reality, cinematography, and education. Recent work on text-guided 3D object generation has shown great promise in addressing these needs. We show that our diffusion-based articulated head avatars outperform state-of-the-art approaches for this task.
  arXiv  Detail & Related papers  (2023-07-10T19:15:32Z)
• AG3D: Learning to Generate 3D Avatars from 2D Image Collections [96.28021214088746]
  We propose a new adversarial generative model of realistic 3D people from 2D images. Our method captures shape and deformation of the body and loose clothing by adopting a holistic 3D generator. We experimentally find that our method outperforms previous 3D- and articulation-aware methods in terms of geometry and appearance.
  arXiv  Detail & Related papers  (2023-05-03T17:56:24Z)
• HoloDiffusion: Training a 3D Diffusion Model using 2D Images [71.1144397510333]
  We introduce a new diffusion setup that can be trained, end-to-end, with only posed 2D images for supervision. We show that our diffusion models are scalable, train robustly, and are competitive in terms of sample quality and fidelity to existing approaches for 3D generative modeling.
  arXiv  Detail & Related papers  (2023-03-29T07:35:56Z)
• Head3D: Complete 3D Head Generation via Tri-plane Feature Distillation [56.267877301135634]
  Current full head generation methods require a large number of 3D scans or multi-view images to train the model. We propose Head3D, a method to generate full 3D heads with limited multi-view images. Our model achieves cost-efficient and diverse complete head generation with photo-realistic renderings and high-quality geometry representations.
  arXiv  Detail & Related papers  (2023-03-28T11:12:26Z)
• GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images [72.15855070133425]
  We introduce GET3D, a Generative model that directly generates Explicit Textured 3D meshes with complex topology, rich geometric details, and high-fidelity textures. GET3D is able to generate high-quality 3D textured meshes, ranging from cars, chairs, animals, motorbikes and human characters to buildings.
  arXiv  Detail & Related papers  (2022-09-22T17:16:19Z)
• Building 3D Generative Models from Minimal Data [3.472931603805115]
  We show that our approach can be used to perform face recognition using only a single 3D template (one scan total, not one per person) We extend our model to a preliminary unsupervised learning framework that enables the learning of the distribution of 3D faces using one 3D template and a small number of 2D images.
  arXiv  Detail & Related papers  (2022-03-04T20:10:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.