TryOnDiffusion: A Tale of Two UNets

• URL: http://arxiv.org/abs/2306.08276v1
• Date: Wed, 14 Jun 2023 06:25:58 GMT
• Title: TryOnDiffusion: A Tale of Two UNets
• Authors: Luyang Zhu, Dawei Yang, Tyler Zhu, Fitsum Reda, William Chan, Chitwan Saharia, Mohammad Norouzi, Ira Kemelmacher-Shlizerman
• Abstract summary: Given two images depicting a person and a garment worn by another person, our goal is to generate a visualization of how the garment might look on the input person. A key challenge is to synthesize a detail-preserving visualization of the garment, while warping the garment to accommodate a significant body pose and shape change. We propose a diffusion-based architecture that unifies two UNets (referred to as Parallel-UNet)
• Score: 46.54704157349114
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Given two images depicting a person and a garment worn by another person, our goal is to generate a visualization of how the garment might look on the input person. A key challenge is to synthesize a photorealistic detail-preserving visualization of the garment, while warping the garment to accommodate a significant body pose and shape change across the subjects. Previous methods either focus on garment detail preservation without effective pose and shape variation, or allow try-on with the desired shape and pose but lack garment details. In this paper, we propose a diffusion-based architecture that unifies two UNets (referred to as Parallel-UNet), which allows us to preserve garment details and warp the garment for significant pose and body change in a single network. The key ideas behind Parallel-UNet include: 1) garment is warped implicitly via a cross attention mechanism, 2) garment warp and person blend happen as part of a unified process as opposed to a sequence of two separate tasks. Experimental results indicate that TryOnDiffusion achieves state-of-the-art performance both qualitatively and quantitatively.

Related papers

• Improving Diffusion Models for Authentic Virtual Try-on in the Wild [53.96244595495942]
  This paper considers image-based virtual try-on, which renders an image of a person wearing a curated garment. We propose a novel diffusion model that improves garment fidelity and generates authentic virtual try-on images. We present a customization method using a pair of person-garment images, which significantly improves fidelity and authenticity.
  arXiv  Detail & Related papers  (2024-03-08T08:12:18Z)
• Learning Garment DensePose for Robust Warping in Virtual Try-On [72.13052519560462]
  We propose a robust warping method for virtual try-on based on a learned garment DensePose. Our method achieves the state-of-the-art equivalent on virtual try-on benchmarks.
  arXiv  Detail & Related papers  (2023-03-30T20:02:29Z)
• DIG: Draping Implicit Garment over the Human Body [56.68349332089129]
  We propose an end-to-end differentiable pipeline that represents garments using implicit surfaces and learns a skinning field conditioned on shape and pose parameters of an articulated body model. We show that our method, thanks to its end-to-end differentiability, allows to recover body and garments parameters jointly from image observations.
  arXiv  Detail & Related papers  (2022-09-22T08:13:59Z)
• Single Stage Virtual Try-on via Deformable Attention Flows [51.70606454288168]
  Virtual try-on aims to generate a photo-realistic fitting result given an in-shop garment and a reference person image. We develop a novel Deformable Attention Flow (DAFlow) which applies the deformable attention scheme to multi-flow estimation. Our proposed method achieves state-of-the-art performance both qualitatively and quantitatively.
  arXiv  Detail & Related papers  (2022-07-19T10:01:31Z)
• Dressing in the Wild by Watching Dance Videos [69.7692630502019]
  This paper attends to virtual try-on in real-world scenes and brings improvements in authenticity and naturalness. We propose a novel generative network called wFlow that can effectively push up garment transfer to in-the-wild context.
  arXiv  Detail & Related papers  (2022-03-29T08:05:45Z)
• Garment4D: Garment Reconstruction from Point Cloud Sequences [12.86951061306046]
  Learning to reconstruct 3D garments is important for dressing 3D human bodies of different shapes in different poses. Previous works typically rely on 2D images as input, which however suffer from the scale and pose ambiguities. We propose a principled framework, Garment4D, that uses 3D point cloud sequences of dressed humans for garment reconstruction.
  arXiv  Detail & Related papers  (2021-12-08T08:15:20Z)
• Per Garment Capture and Synthesis for Real-time Virtual Try-on [15.128477359632262]
  Existing image-based works try to synthesize a try-on image from a single image of a target garment. It is difficult to reproduce the change of wrinkles caused by pose and body size change, as well as pulling and stretching of the garment by hand. We propose an alternative per garment capture and synthesis workflow to handle such rich interactions by training the model with many systematically captured images.
  arXiv  Detail & Related papers  (2021-09-10T03:49:37Z)
• Dressing in Order: Recurrent Person Image Generation for Pose Transfer, Virtual Try-on and Outfit Editing [15.764620091391603]
  This paper proposes a flexible person generation framework called Dressing in Order (DiOr) It supports 2D pose transfer, virtual try-on, and several fashion editing tasks.
  arXiv  Detail & Related papers  (2021-04-14T17:58:54Z)
• VOGUE: Try-On by StyleGAN Interpolation Optimization [14.327659393182204]
  Given an image of a target person and an image of another person wearing a garment, we automatically generate the target garment. At the core of our method is a pose-conditioned StyleGAN2 latent space, which seamlessly combines the areas of interest from each image. Our algorithm allows for garments to deform according to the given body shape, while preserving pattern and material details.
  arXiv  Detail & Related papers  (2021-01-06T22:01:46Z)
• GarmentGAN: Photo-realistic Adversarial Fashion Transfer [0.0]
  GarmentGAN performs image-based garment transfer through generative adversarial methods. The framework allows users to virtually try-on items before purchase and generalizes to various apparel types.
  arXiv  Detail & Related papers  (2020-03-04T05:01:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.