UltrAvatar: A Realistic Animatable 3D Avatar Diffusion Model with Authenticity Guided Textures

• URL: http://arxiv.org/abs/2401.11078v2
• Date: Mon, 30 Sep 2024 02:14:19 GMT
• Title: UltrAvatar: A Realistic Animatable 3D Avatar Diffusion Model with Authenticity Guided Textures
• Authors: Mingyuan Zhou, Rakib Hyder, Ziwei Xuan, Guojun Qi,
• Abstract summary: We propose a novel 3D avatar generation approach termed UltrAvatar with enhanced fidelity of geometry, and superior quality of physically based rendering (PBR) textures without unwanted lighting. We demonstrate the effectiveness and robustness of the proposed method, outperforming the state-of-the-art methods by a large margin in the experiments.
• Score: 80.047065473698
• License:
• Abstract: Recent advances in 3D avatar generation have gained significant attentions. These breakthroughs aim to produce more realistic animatable avatars, narrowing the gap between virtual and real-world experiences. Most of existing works employ Score Distillation Sampling (SDS) loss, combined with a differentiable renderer and text condition, to guide a diffusion model in generating 3D avatars. However, SDS often generates oversmoothed results with few facial details, thereby lacking the diversity compared with ancestral sampling. On the other hand, other works generate 3D avatar from a single image, where the challenges of unwanted lighting effects, perspective views, and inferior image quality make them difficult to reliably reconstruct the 3D face meshes with the aligned complete textures. In this paper, we propose a novel 3D avatar generation approach termed UltrAvatar with enhanced fidelity of geometry, and superior quality of physically based rendering (PBR) textures without unwanted lighting. To this end, the proposed approach presents a diffuse color extraction model and an authenticity guided texture diffusion model. The former removes the unwanted lighting effects to reveal true diffuse colors so that the generated avatars can be rendered under various lighting conditions. The latter follows two gradient-based guidances for generating PBR textures to render diverse face-identity features and details better aligning with 3D mesh geometry. We demonstrate the effectiveness and robustness of the proposed method, outperforming the state-of-the-art methods by a large margin in the experiments.

Related papers

• SEEAvatar: Photorealistic Text-to-3D Avatar Generation with Constrained Geometry and Appearance [37.85026590250023]
  We present SEEAvatar, a method for generating photorealistic 3D avatars from text. For geometry, we propose to constrain the optimized avatar in a decent global shape with a template avatar. For appearance generation, we use diffusion model enhanced by prompt engineering to guide a physically based rendering pipeline.
  arXiv  Detail & Related papers  (2023-12-13T14:48:35Z)
• Reality's Canvas, Language's Brush: Crafting 3D Avatars from Monocular Video [14.140380599168628]
  ReCaLaB is a pipeline that learns high-fidelity 3D human avatars from just a single RGB video. A pose-conditioned NeRF is optimized to volumetrically represent a human subject in canonical T-pose. An image-conditioned diffusion model thereby helps to animate appearance and pose of the 3D avatar to create video sequences with previously unseen human motion.
  arXiv  Detail & Related papers  (2023-12-08T01:53:06Z)
• GETAvatar: Generative Textured Meshes for Animatable Human Avatars [69.56959932421057]
  We study the problem of 3D-aware full-body human generation, aiming at creating animatable human avatars with high-quality geometries and textures. We propose GETAvatar, a Generative model that directly generates Explicit Textured 3D rendering for animatable human Avatar.
  arXiv  Detail & Related papers  (2023-10-04T10:30:24Z)
• Articulated 3D Head Avatar Generation using Text-to-Image Diffusion Models [107.84324544272481]
  The ability to generate diverse 3D articulated head avatars is vital to a plethora of applications, including augmented reality, cinematography, and education. Recent work on text-guided 3D object generation has shown great promise in addressing these needs. We show that our diffusion-based articulated head avatars outperform state-of-the-art approaches for this task.
  arXiv  Detail & Related papers  (2023-07-10T19:15:32Z)
• DreamAvatar: Text-and-Shape Guided 3D Human Avatar Generation via Diffusion Models [55.71306021041785]
  We present DreamAvatar, a text-and-shape guided framework for generating high-quality 3D human avatars. We leverage the SMPL model to provide shape and pose guidance for the generation. We also jointly optimize the losses computed from the full body and from the zoomed-in 3D head to alleviate the common multi-face ''Janus'' problem.
  arXiv  Detail & Related papers  (2023-04-03T12:11:51Z)
• Next3D: Generative Neural Texture Rasterization for 3D-Aware Head Avatars [36.4402388864691]
  3D-aware generative adversarial networks (GANs) synthesize high-fidelity and multi-view-consistent facial images using only collections of single-view 2D imagery. Recent efforts incorporate 3D Morphable Face Model (3DMM) to describe deformation in generative radiance fields either explicitly or implicitly. We propose a novel 3D GAN framework for unsupervised learning of generative, high-quality and 3D-consistent facial avatars from unstructured 2D images.
  arXiv  Detail & Related papers  (2022-11-21T06:40:46Z)
• DRaCoN -- Differentiable Rasterization Conditioned Neural Radiance Fields for Articulated Avatars [92.37436369781692]
  We present DRaCoN, a framework for learning full-body volumetric avatars. It exploits the advantages of both the 2D and 3D neural rendering techniques. Experiments on the challenging ZJU-MoCap and Human3.6M datasets indicate that DRaCoN outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2022-03-29T17:59:15Z)
• AvatarMe++: Facial Shape and BRDF Inference with Photorealistic Rendering-Aware GANs [119.23922747230193]
  We introduce the first method that is able to reconstruct render-ready 3D facial geometry and BRDF from a single "in-the-wild" image. Our method outperforms the existing arts by a significant margin and reconstructs high-resolution 3D faces from a single low-resolution image.
  arXiv  Detail & Related papers  (2021-12-11T11:36:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.