DH-VTON: Deep Text-Driven Virtual Try-On via Hybrid Attention Learning

• URL: http://arxiv.org/abs/2410.12501v1
• Date: Wed, 16 Oct 2024 12:27:10 GMT
• Title: DH-VTON: Deep Text-Driven Virtual Try-On via Hybrid Attention Learning
• Authors: Jiabao Wei, Zhiyuan Ma,
• Abstract summary: DH-VTON is a deep text-driven virtual try-on model featuring a special hybrid attention learning strategy and deep garment semantic preservation module. To extract the deep semantics of the garments, we first introduce InternViT-6B as fine-grained feature learner, which can be trained to align with the large-scale intrinsic knowledge. To enhance the customized dressing abilities, we further introduce Garment-Feature ControlNet Plus (abbr. GFC+) module.
• Score: 6.501730122478447
• License:
• Abstract: Virtual Try-ON (VTON) aims to synthesis specific person images dressed in given garments, which recently receives numerous attention in online shopping scenarios. Currently, the core challenges of the VTON task mainly lie in the fine-grained semantic extraction (i.e.,deep semantics) of the given reference garments during depth estimation and effective texture preservation when the garments are synthesized and warped onto human body. To cope with these issues, we propose DH-VTON, a deep text-driven virtual try-on model featuring a special hybrid attention learning strategy and deep garment semantic preservation module. By standing on the shoulder of a well-built pre-trained paint-by-example (abbr. PBE) approach, we present our DH-VTON pipeline in this work. Specifically, to extract the deep semantics of the garments, we first introduce InternViT-6B as fine-grained feature learner, which can be trained to align with the large-scale intrinsic knowledge with deep text semantics (e.g.,"neckline" or "girdle") to make up for the deficiency of the commonly adopted CLIP encoder. Based on this, to enhance the customized dressing abilities, we further introduce Garment-Feature ControlNet Plus (abbr. GFC+) module and propose to leverage a fresh hybrid attention strategy for training, which can adaptively integrate fine-grained characteristics of the garments into the different layers of the VTON model, so as to achieve multi-scale features preservation effects. Extensive experiments on several representative datasets demonstrate that our method outperforms previous diffusion-based and GAN-based approaches, showing competitive performance in preserving garment details and generating authentic human images.

Related papers

• Improving Virtual Try-On with Garment-focused Diffusion Models [91.95830983115474]
  Diffusion models have led to the revolutionizing of generative modeling in numerous image synthesis tasks. We shape a new Diffusion model, namely GarDiff, which triggers the garment-focused diffusion process. Experiments on VITON-HD and DressCode datasets demonstrate the superiority of our GarDiff when compared to state-of-the-art VTON approaches.
  arXiv  Detail & Related papers  (2024-09-12T17:55:11Z)
• IMAGDressing-v1: Customizable Virtual Dressing [58.44155202253754]
  IMAGDressing-v1 is a virtual dressing task that generates freely editable human images with fixed garments and optional conditions. IMAGDressing-v1 incorporates a garment UNet that captures semantic features from CLIP and texture features from VAE. We present a hybrid attention module, including a frozen self-attention and a trainable cross-attention, to integrate garment features from the garment UNet into a frozen denoising UNet.
  arXiv  Detail & Related papers  (2024-07-17T16:26:30Z)
• GraVITON: Graph based garment warping with attention guided inversion for Virtual-tryon [5.790630195329777]
  We introduce a novel graph based warping technique which emphasizes the value of context in garment flow. Our method, validated on VITON-HD and Dresscode datasets, showcases substantial improvement in garment warping, texture preservation, and overall realism.
  arXiv  Detail & Related papers  (2024-06-04T10:29:18Z)
• Improving Diffusion Models for Authentic Virtual Try-on in the Wild [53.96244595495942]
  This paper considers image-based virtual try-on, which renders an image of a person wearing a curated garment. We propose a novel diffusion model that improves garment fidelity and generates authentic virtual try-on images. We present a customization method using a pair of person-garment images, which significantly improves fidelity and authenticity.
  arXiv  Detail & Related papers  (2024-03-08T08:12:18Z)
• OOTDiffusion: Outfitting Fusion based Latent Diffusion for Controllable Virtual Try-on [7.46772222515689]
  OOTDiffusion is a novel network architecture for realistic and controllable image-based virtual try-on. We leverage the power of pretrained latent diffusion models, designing an outfitting UNet to learn the garment detail features. Our experiments on the VITON-HD and Dress Code datasets demonstrate that OOTDiffusion efficiently generates high-quality try-on results.
  arXiv  Detail & Related papers  (2024-03-04T07:17:44Z)
• Harnessing Diffusion Models for Visual Perception with Meta Prompts [68.78938846041767]
  We propose a simple yet effective scheme to harness a diffusion model for visual perception tasks. We introduce learnable embeddings (meta prompts) to the pre-trained diffusion models to extract proper features for perception. Our approach achieves new performance records in depth estimation tasks on NYU depth V2 and KITTI, and in semantic segmentation task on CityScapes.
  arXiv  Detail & Related papers  (2023-12-22T14:40:55Z)
• Single Stage Warped Cloth Learning and Semantic-Contextual Attention Feature Fusion for Virtual TryOn [5.790630195329777]
  Image-based virtual try-on aims to fit an in-shop garment onto a clothed person image. Garment warping, which aligns the target garment with the corresponding body parts in the person image, is a crucial step in achieving this goal. We propose a novel single-stage framework that implicitly learns the same without explicit multi-stage learning.
  arXiv  Detail & Related papers  (2023-10-08T06:05:01Z)
• LaDI-VTON: Latent Diffusion Textual-Inversion Enhanced Virtual Try-On [35.4056826207203]
  This work introduces LaDI-VTON, the first Latent Diffusion textual Inversion-enhanced model for the Virtual Try-ON task. The proposed architecture relies on a latent diffusion model extended with a novel additional autoencoder module. We show that our approach outperforms the competitors by a consistent margin, achieving a significant milestone for the task.
  arXiv  Detail & Related papers  (2023-05-22T21:38:06Z)
• PASTA-GAN++: A Versatile Framework for High-Resolution Unpaired Virtual Try-on [70.12285433529998]
  PASTA-GAN++ is a versatile system for high-resolution unpaired virtual try-on. It supports unsupervised training, arbitrary garment categories, and controllable garment editing.
  arXiv  Detail & Related papers  (2022-07-27T11:47:49Z)
• Towards Scalable Unpaired Virtual Try-On via Patch-Routed Spatially-Adaptive GAN [66.3650689395967]
  We propose a texture-preserving end-to-end network, the PAtch-routed SpaTially-Adaptive GAN (PASTA-GAN), that facilitates real-world unpaired virtual try-on. To disentangle the style and spatial information of each garment, PASTA-GAN consists of an innovative patch-routed disentanglement module.
  arXiv  Detail & Related papers  (2021-11-20T08:36:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.