AvatarStudio: High-fidelity and Animatable 3D Avatar Creation from Text

• URL: http://arxiv.org/abs/2311.17917v1
• Date: Wed, 29 Nov 2023 18:59:32 GMT
• Title: AvatarStudio: High-fidelity and Animatable 3D Avatar Creation from Text
• Authors: Jianfeng Zhang, Xuanmeng Zhang, Huichao Zhang, Jun Hao Liew, Chenxu Zhang, Yi Yang, Jiashi Feng
• Abstract summary: AvatarStudio is a coarse-to-fine generative model that generates explicit textured 3D meshes for animatable human avatars. By effectively leveraging the synergy between the articulated mesh representation and the DensePose-conditional diffusion model, AvatarStudio can create high-quality avatars.
• Score: 71.09533176800707
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We study the problem of creating high-fidelity and animatable 3D avatars from only textual descriptions. Existing text-to-avatar methods are either limited to static avatars which cannot be animated or struggle to generate animatable avatars with promising quality and precise pose control. To address these limitations, we propose AvatarStudio, a coarse-to-fine generative model that generates explicit textured 3D meshes for animatable human avatars. Specifically, AvatarStudio begins with a low-resolution NeRF-based representation for coarse generation, followed by incorporating SMPL-guided articulation into the explicit mesh representation to support avatar animation and high resolution rendering. To ensure view consistency and pose controllability of the resulting avatars, we introduce a 2D diffusion model conditioned on DensePose for Score Distillation Sampling supervision. By effectively leveraging the synergy between the articulated mesh representation and the DensePose-conditional diffusion model, AvatarStudio can create high-quality avatars from text that are ready for animation, significantly outperforming previous methods. Moreover, it is competent for many applications, e.g., multimodal avatar animations and style-guided avatar creation. For more results, please refer to our project page: http://jeff95.me/projects/avatarstudio.html

Related papers

• DreamWaltz-G: Expressive 3D Gaussian Avatars from Skeleton-Guided 2D Diffusion [69.67970568012599]
  We present DreamWaltz-G, a novel learning framework for animatable 3D avatar generation from text. The core of this framework lies in Score Distillation and Hybrid 3D Gaussian Avatar representation. Our framework further supports diverse applications, including human video reenactment and multi-subject scene composition.
  arXiv  Detail & Related papers  (2024-09-25T17:59:45Z)
• WildAvatar: Web-scale In-the-wild Video Dataset for 3D Avatar Creation [55.85887047136534]
  WildAvatar is a web-scale in-the-wild human avatar creation dataset extracted from YouTube. We evaluate several state-of-the-art avatar creation methods on our dataset, highlighting the unexplored challenges in real-world applications on avatar creation.
  arXiv  Detail & Related papers  (2024-07-02T11:17:48Z)
• AniArtAvatar: Animatable 3D Art Avatar from a Single Image [0.0]
  We present a novel approach for generating animatable 3D-aware art avatars from a single image. We use a view-conditioned 2D diffusion model to synthesize multi-view images from a single art portrait with a neutral expression. For avatar animation, we extract control points, transfer the motion with these points, and deform the implicit canonical space.
  arXiv  Detail & Related papers  (2024-03-26T12:08:04Z)
• DivAvatar: Diverse 3D Avatar Generation with a Single Prompt [95.9978722953278]
  DivAvatar is a framework that generates diverse avatars from a single text prompt. It has two key designs that help achieve generation diversity and visual quality. Extensive experiments show that DivAvatar is highly versatile in generating avatars of diverse appearances.
  arXiv  Detail & Related papers  (2024-02-27T08:10:31Z)
• AvatarVerse: High-quality & Stable 3D Avatar Creation from Text and Pose [23.76390935089982]
  We present AvatarVerse, a stable pipeline for generating high expressivequality 3D avatars from text descriptions and pose guidance. To this end, we propose zero-fidelity 3D modeling of 3D avatars that are not only more expressive, but also higher quality stablizes.
  arXiv  Detail & Related papers  (2023-08-07T14:09:46Z)
• AvatarBooth: High-Quality and Customizable 3D Human Avatar Generation [14.062402203105712]
  AvatarBooth is a novel method for generating high-quality 3D avatars using text prompts or specific images. Our key contribution is the precise avatar generation control by using dual fine-tuned diffusion models. We present a multi-resolution rendering strategy that facilitates coarse-to-fine supervision of 3D avatar generation.
  arXiv  Detail & Related papers  (2023-06-16T14:18:51Z)
• DreamWaltz: Make a Scene with Complex 3D Animatable Avatars [68.49935994384047]
  We present DreamWaltz, a novel framework for generating and animating complex 3D avatars given text guidance and parametric human body prior. For animation, our method learns an animatable 3D avatar representation from abundant image priors of diffusion model conditioned on various poses.
  arXiv  Detail & Related papers  (2023-05-21T17:59:39Z)
• SwiftAvatar: Efficient Auto-Creation of Parameterized Stylized Character on Arbitrary Avatar Engines [34.645129752596915]
  We propose SwiftAvatar, a novel avatar auto-creation framework. We synthesize data in high-quality as many as possible, consisting of avatar vectors and their corresponding realistic faces. Our experiments demonstrate the effectiveness and efficiency of SwiftAvatar on two different avatar engines.
  arXiv  Detail & Related papers  (2023-01-19T16:14:28Z)
• StylePeople: A Generative Model of Fullbody Human Avatars [59.42166744151461]
  We propose a new type of full-body human avatars, which combines parametric mesh-based body model with a neural texture. We show that such avatars can successfully model clothing and hair, which usually poses a problem for mesh-based approaches. We then propose a generative model for such avatars that can be trained from datasets of images and videos of people.
  arXiv  Detail & Related papers  (2021-04-16T20:43:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.