DreamWaltz: Make a Scene with Complex 3D Animatable Avatars
- URL: http://arxiv.org/abs/2305.12529v3
- Date: Mon, 6 Nov 2023 03:34:07 GMT
- Title: DreamWaltz: Make a Scene with Complex 3D Animatable Avatars
- Authors: Yukun Huang, Jianan Wang, Ailing Zeng, He Cao, Xianbiao Qi, Yukai Shi,
Zheng-Jun Zha, Lei Zhang
- Abstract summary: We present DreamWaltz, a novel framework for generating and animating complex 3D avatars given text guidance and parametric human body prior.
For animation, our method learns an animatable 3D avatar representation from abundant image priors of diffusion model conditioned on various poses.
- Score: 68.49935994384047
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: We present DreamWaltz, a novel framework for generating and animating complex
3D avatars given text guidance and parametric human body prior. While recent
methods have shown encouraging results for text-to-3D generation of common
objects, creating high-quality and animatable 3D avatars remains challenging.
To create high-quality 3D avatars, DreamWaltz proposes 3D-consistent
occlusion-aware Score Distillation Sampling (SDS) to optimize implicit neural
representations with canonical poses. It provides view-aligned supervision via
3D-aware skeleton conditioning which enables complex avatar generation without
artifacts and multiple faces. For animation, our method learns an animatable 3D
avatar representation from abundant image priors of diffusion model conditioned
on various poses, which could animate complex non-rigged avatars given
arbitrary poses without retraining. Extensive evaluations demonstrate that
DreamWaltz is an effective and robust approach for creating 3D avatars that can
take on complex shapes and appearances as well as novel poses for animation.
The proposed framework further enables the creation of complex scenes with
diverse compositions, including avatar-avatar, avatar-object and avatar-scene
interactions. See https://dreamwaltz3d.github.io/ for more vivid 3D avatar and
animation results.
Related papers
- DreamWaltz-G: Expressive 3D Gaussian Avatars from Skeleton-Guided 2D
Diffusion [69.67970568012599]
We present DreamWaltz-G, a novel learning framework for animatable 3D avatar generation from text.
The core of this framework lies in Score Distillation and Hybrid 3D Gaussian Avatar representation.
Our framework further supports diverse applications, including human video reenactment and multi-subject scene composition.
arXiv Detail & Related papers (2024-09-25T17:59:45Z) - AniArtAvatar: Animatable 3D Art Avatar from a Single Image [0.0]
We present a novel approach for generating animatable 3D-aware art avatars from a single image.
We use a view-conditioned 2D diffusion model to synthesize multi-view images from a single art portrait with a neutral expression.
For avatar animation, we extract control points, transfer the motion with these points, and deform the implicit canonical space.
arXiv Detail & Related papers (2024-03-26T12:08:04Z) - AvatarStudio: High-fidelity and Animatable 3D Avatar Creation from Text [71.09533176800707]
AvatarStudio is a coarse-to-fine generative model that generates explicit textured 3D meshes for animatable human avatars.
By effectively leveraging the synergy between the articulated mesh representation and the DensePose-conditional diffusion model, AvatarStudio can create high-quality avatars.
arXiv Detail & Related papers (2023-11-29T18:59:32Z) - AvatarVerse: High-quality & Stable 3D Avatar Creation from Text and Pose [23.76390935089982]
We present AvatarVerse, a stable pipeline for generating high expressivequality 3D avatars from text descriptions and pose guidance.
To this end, we propose zero-fidelity 3D modeling of 3D avatars that are not only more expressive, but also higher quality stablizes.
arXiv Detail & Related papers (2023-08-07T14:09:46Z) - DreamAvatar: Text-and-Shape Guided 3D Human Avatar Generation via
Diffusion Models [55.71306021041785]
We present DreamAvatar, a text-and-shape guided framework for generating high-quality 3D human avatars.
We leverage the SMPL model to provide shape and pose guidance for the generation.
We also jointly optimize the losses computed from the full body and from the zoomed-in 3D head to alleviate the common multi-face ''Janus'' problem.
arXiv Detail & Related papers (2023-04-03T12:11:51Z) - AvatarGen: A 3D Generative Model for Animatable Human Avatars [108.11137221845352]
AvatarGen is an unsupervised generation of 3D-aware clothed humans with various appearances and controllable geometries.
Our method can generate animatable 3D human avatars with high-quality appearance and geometry modeling.
It is competent for many applications, e.g., single-view reconstruction, re-animation, and text-guided synthesis/editing.
arXiv Detail & Related papers (2022-11-26T15:15:45Z) - AvatarGen: a 3D Generative Model for Animatable Human Avatars [108.11137221845352]
AvatarGen is the first method that enables not only non-rigid human generation with diverse appearance but also full control over poses and viewpoints.
To model non-rigid dynamics, it introduces a deformation network to learn pose-dependent deformations in the canonical space.
Our method can generate animatable human avatars with high-quality appearance and geometry modeling, significantly outperforming previous 3D GANs.
arXiv Detail & Related papers (2022-08-01T01:27:02Z) - AvatarCLIP: Zero-Shot Text-Driven Generation and Animation of 3D Avatars [37.43588165101838]
AvatarCLIP is a zero-shot text-driven framework for 3D avatar generation and animation.
We take advantage of the powerful vision-language model CLIP for supervising neural human generation.
By leveraging the priors learned in the motion VAE, a CLIP-guided reference-based motion synthesis method is proposed for the animation of the generated 3D avatar.
arXiv Detail & Related papers (2022-05-17T17:59:19Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.