ClotheDreamer: Text-Guided Garment Generation with 3D Gaussians

• URL: http://arxiv.org/abs/2406.16815v1
• Date: Mon, 24 Jun 2024 17:25:39 GMT
• Title: ClotheDreamer: Text-Guided Garment Generation with 3D Gaussians
• Authors: Yufei Liu, Junshu Tang, Chu Zheng, Shijie Zhang, Jinkun Hao, Junwei Zhu, Dongjin Huang,
• Abstract summary: ClotheDreamer is a method for generating wearable, production-ready 3D garment assets from text prompts. We propose a novel representation Disentangled Clothe Gaussian Splatting (DCGS) to enable separate optimization.
• Score: 13.196912161879936
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: High-fidelity 3D garment synthesis from text is desirable yet challenging for digital avatar creation. Recent diffusion-based approaches via Score Distillation Sampling (SDS) have enabled new possibilities but either intricately couple with human body or struggle to reuse. We introduce ClotheDreamer, a 3D Gaussian-based method for generating wearable, production-ready 3D garment assets from text prompts. We propose a novel representation Disentangled Clothe Gaussian Splatting (DCGS) to enable separate optimization. DCGS represents clothed avatar as one Gaussian model but freezes body Gaussian splats. To enhance quality and completeness, we incorporate bidirectional SDS to supervise clothed avatar and garment RGBD renderings respectively with pose conditions and propose a new pruning strategy for loose clothing. Our approach can also support custom clothing templates as input. Benefiting from our design, the synthetic 3D garment can be easily applied to virtual try-on and support physically accurate animation. Extensive experiments showcase our method's superior and competitive performance. Our project page is at https://ggxxii.github.io/clothedreamer.

Related papers

• DAGSM: Disentangled Avatar Generation with GS-enhanced Mesh [102.84518904896737]
  DAGSM is a novel pipeline that generates disentangled human bodies and garments from the given text prompts. We first create the unclothed body, followed by a sequence of individual cloth generation based on the body. Experiments have demonstrated that DAGSM generates high-quality disentangled avatars, supports clothing replacement and realistic animation, and outperforms the baselines in visual quality.
  arXiv  Detail & Related papers  (2024-11-20T07:00:48Z)
• Generalizable and Animatable Gaussian Head Avatar [50.34788590904843]
  We propose Generalizable and Animatable Gaussian head Avatar (GAGAvatar) for one-shot animatable head avatar reconstruction. We generate the parameters of 3D Gaussians from a single image in a single forward pass. Our method exhibits superior performance compared to previous methods in terms of reconstruction quality and expression accuracy.
  arXiv  Detail & Related papers  (2024-10-10T14:29:00Z)
• DreamWaltz-G: Expressive 3D Gaussian Avatars from Skeleton-Guided 2D Diffusion [69.67970568012599]
  We present DreamWaltz-G, a novel learning framework for animatable 3D avatar generation from text. The core of this framework lies in Score Distillation and Hybrid 3D Gaussian Avatar representation. Our framework further supports diverse applications, including human video reenactment and multi-subject scene composition.
  arXiv  Detail & Related papers  (2024-09-25T17:59:45Z)
• HumanCoser: Layered 3D Human Generation via Semantic-Aware Diffusion Model [43.66218796152962]
  This paper aims to generate physically-layered 3D humans from text prompts. We propose a novel layer-wise dressed human representation based on a physically-decoupled diffusion model. To match the clothing with different body shapes, we propose an SMPL-driven implicit field network.
  arXiv  Detail & Related papers  (2024-08-21T06:00:11Z)
• GarmentDreamer: 3DGS Guided Garment Synthesis with Diverse Geometry and Texture Details [31.92583566128599]
  Traditional 3D garment creation is labor-intensive, involving sketching, modeling, UV mapping, and time-consuming processes. We propose GarmentDreamer, a novel method that leverages 3D Gaussian Splatting (GS) as guidance to generate 3D garment from text prompts.
  arXiv  Detail & Related papers  (2024-05-20T23:54:28Z)
• Garment3DGen: 3D Garment Stylization and Texture Generation [11.836357439129301]
  Garment3DGen is a new method to synthesize 3D garment assets from a base mesh given a single input image as guidance. We leverage the recent progress of image-to-3D diffusion methods to generate 3D garment geometries. We generate high-fidelity texture maps that are globally and locally consistent and faithfully capture the input guidance.
  arXiv  Detail & Related papers  (2024-03-27T17:59:33Z)
• Layered 3D Human Generation via Semantic-Aware Diffusion Model [63.459666003261276]
  We propose a text-driven layered 3D human generation framework based on a novel semantic-aware diffusion model. To keep the generated clothing consistent with the target text, we propose a semantic-confidence strategy for clothing. To match the clothing with different body shapes, we propose a SMPL-driven implicit field deformation network.
  arXiv  Detail & Related papers  (2023-12-10T07:34:43Z)
• PERGAMO: Personalized 3D Garments from Monocular Video [6.8338761008826445]
  PERGAMO is a data-driven approach to learn a deformable model for 3D garments from monocular images. We first introduce a novel method to reconstruct the 3D geometry of garments from a single image, and use it to build a dataset of clothing from monocular videos. We show that our method is capable of producing garment animations that match the real-world behaviour, and generalizes to unseen body motions extracted from motion capture dataset.
  arXiv  Detail & Related papers  (2022-10-26T21:15:54Z)
• Capturing and Animation of Body and Clothing from Monocular Video [105.87228128022804]
  We present SCARF, a hybrid model combining a mesh-based body with a neural radiance field. integrating the mesh into the rendering enables us to optimize SCARF directly from monocular videos. We demonstrate that SCARFs clothing with higher visual quality than existing methods, that the clothing deforms with changing body pose and body shape, and that clothing can be successfully transferred between avatars of different subjects.
  arXiv  Detail & Related papers  (2022-10-04T19:34:05Z)
• The Power of Points for Modeling Humans in Clothing [60.00557674969284]
  Currently it requires an artist to create 3D human avatars with realistic clothing that can move naturally. We show that a 3D representation can capture varied topology at high resolution and that can be learned from data. We train a neural network with a novel local clothing geometric feature to represent the shape of different outfits.
  arXiv  Detail & Related papers  (2021-09-02T17:58:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.