DAGSM: Disentangled Avatar Generation with GS-enhanced Mesh

• URL: http://arxiv.org/abs/2411.15205v1
• Date: Wed, 20 Nov 2024 07:00:48 GMT
• Title: DAGSM: Disentangled Avatar Generation with GS-enhanced Mesh
• Authors: Jingyu Zhuang, Di Kang, Linchao Bao, Liang Lin, Guanbin Li,
• Abstract summary: DAGSM is a novel pipeline that generates disentangled human bodies and garments from the given text prompts. We first create the unclothed body, followed by a sequence of individual cloth generation based on the body. Experiments have demonstrated that DAGSM generates high-quality disentangled avatars, supports clothing replacement and realistic animation, and outperforms the baselines in visual quality.
• Score: 102.84518904896737
• License:
• Abstract: Text-driven avatar generation has gained significant attention owing to its convenience. However, existing methods typically model the human body with all garments as a single 3D model, limiting its usability, such as clothing replacement, and reducing user control over the generation process. To overcome the limitations above, we propose DAGSM, a novel pipeline that generates disentangled human bodies and garments from the given text prompts. Specifically, we model each part (e.g., body, upper/lower clothes) of the clothed human as one GS-enhanced mesh (GSM), which is a traditional mesh attached with 2D Gaussians to better handle complicated textures (e.g., woolen, translucent clothes) and produce realistic cloth animations. During the generation, we first create the unclothed body, followed by a sequence of individual cloth generation based on the body, where we introduce a semantic-based algorithm to achieve better human-cloth and garment-garment separation. To improve texture quality, we propose a view-consistent texture refinement module, including a cross-view attention mechanism for texture style consistency and an incident-angle-weighted denoising (IAW-DE) strategy to update the appearance. Extensive experiments have demonstrated that DAGSM generates high-quality disentangled avatars, supports clothing replacement and realistic animation, and outperforms the baselines in visual quality.

Related papers

• PICA: Physics-Integrated Clothed Avatar [30.277983921620663]
  We introduce PICA, a novel representation for high-fidelity animatable clothed human avatars with physics-accurate dynamics, even for loose clothing. Our method achieves high-fidelity rendering of human bodies in complex and novel driving poses, significantly outperforming previous methods under the same settings.
  arXiv  Detail & Related papers  (2024-07-07T10:23:21Z)
• ClotheDreamer: Text-Guided Garment Generation with 3D Gaussians [13.196912161879936]
  ClotheDreamer is a method for generating wearable, production-ready 3D garment assets from text prompts. We propose a novel representation Disentangled Clothe Gaussian Splatting (DCGS) to enable separate optimization.
  arXiv  Detail & Related papers  (2024-06-24T17:25:39Z)
• TELA: Text to Layer-wise 3D Clothed Human Generation [27.93447899876341]
  This paper addresses the task of 3D clothed human generation from textural descriptions. We propose a layer-wise clothed human representation combined with a progressive optimization strategy. Our approach achieves state-of-the-art 3D clothed human generation while also supporting cloth editing applications.
  arXiv  Detail & Related papers  (2024-04-25T17:05:38Z)
• DressCode: Autoregressively Sewing and Generating Garments from Text Guidance [61.48120090970027]
  DressCode aims to democratize design for novices and offer immense potential in fashion design, virtual try-on, and digital human creation. We first introduce SewingGPT, a GPT-based architecture integrating cross-attention with text-conditioned embedding to generate sewing patterns. We then tailor a pre-trained Stable Diffusion to generate tile-based Physically-based Rendering (PBR) textures for the garments.
  arXiv  Detail & Related papers  (2024-01-29T16:24:21Z)
• AniDress: Animatable Loose-Dressed Avatar from Sparse Views Using Garment Rigging Model [58.035758145894846]
  We introduce AniDress, a novel method for generating animatable human avatars in loose clothes using very sparse multi-view videos. A pose-driven deformable neural radiance field conditioned on both body and garment motions is introduced, providing explicit control of both parts. Our method is able to render natural garment dynamics that deviate highly from the body and well to generalize to both unseen views and poses.
  arXiv  Detail & Related papers  (2024-01-27T08:48:18Z)
• Layered 3D Human Generation via Semantic-Aware Diffusion Model [63.459666003261276]
  We propose a text-driven layered 3D human generation framework based on a novel semantic-aware diffusion model. To keep the generated clothing consistent with the target text, we propose a semantic-confidence strategy for clothing. To match the clothing with different body shapes, we propose a SMPL-driven implicit field deformation network.
  arXiv  Detail & Related papers  (2023-12-10T07:34:43Z)
• Disentangled Clothed Avatar Generation from Text Descriptions [41.01453534915251]
  We introduce a novel text-to-avatar generation method that separately generates the human body and the clothes. Our approach achieves higher texture and geometry quality and better semantic alignment with text prompts.
  arXiv  Detail & Related papers  (2023-12-08T18:43:12Z)
• XAGen: 3D Expressive Human Avatars Generation [76.69560679209171]
  XAGen is the first 3D generative model for human avatars capable of expressive control over body, face, and hands. We propose a multi-part rendering technique that disentangles the synthesis of body, face, and hands. Experiments show that XAGen surpasses state-of-the-art methods in terms of realism, diversity, and expressive control abilities.
  arXiv  Detail & Related papers  (2023-11-22T18:30:42Z)
• Text-guided 3D Human Generation from 2D Collections [69.04031635550294]
  We introduce Text-guided 3D Human Generation (texttT3H), where a model is to generate a 3D human, guided by the fashion description. CCH adopts cross-modal attention to fuse compositional human rendering with the extracted fashion semantics. We conduct evaluations on DeepFashion and SHHQ with diverse fashion attributes covering the shape, fabric, and color of upper and lower clothing.
  arXiv  Detail & Related papers  (2023-05-23T17:50:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.