PEMF-VTO: Point-Enhanced Video Virtual Try-on via Mask-free Paradigm

• URL: http://arxiv.org/abs/2412.03021v4
• Date: Fri, 14 Mar 2025 10:07:40 GMT
• Title: PEMF-VTO: Point-Enhanced Video Virtual Try-on via Mask-free Paradigm
• Authors: Tianyu Chang, Xiaohao Chen, Zhichao Wei, Xuanpu Zhang, Qing-Guo Chen, Weihua Luo, Peipei Song, Xun Yang,
• Abstract summary: Video Virtual Try-on aims to seamlessly transfer a reference garment onto a target person in a video.<n>Existing methods typically rely on inpainting masks to define the try-on area.<n>We propose PEMF-VTO, a novel Point-Enhanced Mask-Free Video Virtual Try-On framework.
• Score: 21.1235226974745
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Video Virtual Try-on aims to seamlessly transfer a reference garment onto a target person in a video while preserving both visual fidelity and temporal coherence. Existing methods typically rely on inpainting masks to define the try-on area, enabling accurate garment transfer for simple scenes (e.g., in-shop videos). However, these mask-based approaches struggle with complex real-world scenarios, as overly large and inconsistent masks often destroy spatial-temporal information, leading to distorted results. Mask-free methods alleviate this issue but face challenges in accurately determining the try-on area, especially for videos with dynamic body movements. To address these limitations, we propose PEMF-VTO, a novel Point-Enhanced Mask-Free Video Virtual Try-On framework that leverages sparse point alignments to explicitly guide garment transfer. Our key innovation is the introduction of point-enhanced guidance, which provides flexible and reliable control over both spatial-level garment transfer and temporal-level video coherence. Specifically, we design a Point-Enhanced Transformer (PET) with two core components: Point-Enhanced Spatial Attention (PSA), which uses frame-cloth point alignments to precisely guide garment transfer, and Point-Enhanced Temporal Attention (PTA), which leverages frame-frame point correspondences to enhance temporal coherence and ensure smooth transitions across frames. Extensive experiments demonstrate that our PEMF-VTO outperforms state-of-the-art methods, generating more natural, coherent, and visually appealing try-on videos, particularly for challenging in-the-wild scenarios. The link to our paper's homepage is https://pemf-vto.github.io/.

Related papers

• Video Virtual Try-on with Conditional Diffusion Transformer Inpainter [27.150975905047968]
  Video virtual try-on aims to fit a garment to a target person in consecutive video frames.<n>Recent diffusion-based video try-on methods, though very few, happen to coincide with a similar solution.<n>We propose ViTI (Video Try-on Inpainter), formulate and implement video virtual try-on as a conditional video inpainting task.
  arXiv  Detail & Related papers  (2025-06-26T13:56:27Z)
• MagicTryOn: Harnessing Diffusion Transformer for Garment-Preserving Video Virtual Try-on [16.0505428363005]
  We propose MagicTryOn, a video virtual try-on framework built upon the large-scale video diffusion Transformer.<n>We replace the U-Net architecture with a diffusion Transformer and combine full self-attention to model the garment consistency of videos.<n>Our method outperforms existing SOTA methods in comprehensive evaluations and generalizes to in-the-wild scenarios.
  arXiv  Detail & Related papers  (2025-05-27T15:22:02Z)
• 3DV-TON: Textured 3D-Guided Consistent Video Try-on via Diffusion Models [12.949009540192389]
  3DV-TON is a novel framework for generating high-fidelity and temporally consistent video try-on results. Our approach employs generated animatable textured 3D meshes as explicit frame-level guidance. To advance video try-on research, we introduce HR-VVT, a high-resolution benchmark dataset containing 130 videos with diverse clothing types and scenarios.
  arXiv  Detail & Related papers  (2025-04-24T10:12:40Z)
• VideoAnydoor: High-fidelity Video Object Insertion with Precise Motion Control [66.66226299852559]
  VideoAnydoor is a zero-shot video object insertion framework with high-fidelity detail preservation and precise motion control. To preserve the detailed appearance and meanwhile support fine-grained motion control, we design a pixel warper.
  arXiv  Detail & Related papers  (2025-01-02T18:59:54Z)
• Replace Anyone in Videos [82.37852750357331]
  We present the ReplaceAnyone framework, which focuses on localized human replacement and insertion featuring intricate backgrounds.<n>We formulate this task as an image-conditioned video inpainting paradigm with pose guidance, utilizing a unified end-to-end video diffusion architecture.<n>The proposed ReplaceAnyone can be seamlessly applied not only to traditional 3D-UNet base models but also to DiT-based video models such as Wan2.1.
  arXiv  Detail & Related papers  (2024-09-30T03:27:33Z)
• Blended Latent Diffusion under Attention Control for Real-World Video Editing [5.659933808910005]
  We propose to adapt a image-level blended latent diffusion model to perform local video editing tasks. Specifically, we leverage DDIM inversion to acquire the latents as background latents instead of the randomly noised ones. We also introduce an autonomous mask manufacture mechanism derived from cross-attention maps in diffusion steps.
  arXiv  Detail & Related papers  (2024-09-05T13:23:52Z)
• Text-Guided Video Masked Autoencoder [12.321239366215426]
  We introduce a novel text-guided masking algorithm (TGM) that masks the video regions with highest correspondence to paired captions. We show that across existing masking algorithms, unifying MAE and masked video-text contrastive learning improves downstream performance compared to pure MAE.
  arXiv  Detail & Related papers  (2024-08-01T17:58:19Z)
• WildVidFit: Video Virtual Try-On in the Wild via Image-Based Controlled Diffusion Models [132.77237314239025]
  Video virtual try-on aims to generate realistic sequences that maintain garment identity and adapt to a person's pose and body shape in source videos. Traditional image-based methods, relying on warping and blending, struggle with complex human movements and occlusions. We reconceptualize video try-on as a process of generating videos conditioned on garment descriptions and human motion. Our solution, WildVidFit, employs image-based controlled diffusion models for a streamlined, one-stage approach.
  arXiv  Detail & Related papers  (2024-07-15T11:21:03Z)
• Drag-A-Video: Non-rigid Video Editing with Point-based Interaction [63.78538355189017]
  We propose a new diffusion-based method for interactive point-based video manipulation, called Drag-A-Video. Our method allows users to click pairs of handle points and target points as well as masks on the first frame of an input video. To precisely modify the contents of the video, we employ a new video-level motion supervision to update the features of the video.
  arXiv  Detail & Related papers  (2023-12-05T18:05:59Z)
• Unsupervised Region-Growing Network for Object Segmentation in Atmospheric Turbulence [10.8380383565446]
  We present an unsupervised approach for segmenting moving objects in videos downgraded by atmospheric turbulence. We first identify a small set of moving object pixels with high confidence, then gradually grow a foreground mask from those seeds to segment all moving objects. Results show that our method achieves good accuracy in segmenting moving objects and is robust for long-range videos with various turbulence strengths.
  arXiv  Detail & Related papers  (2023-11-06T22:17:18Z)
• Siamese Masked Autoencoders [76.35448665609998]
  We present Siamese Masked Autoencoders (SiamMAE) for learning visual correspondence from videos. SiamMAE operates on pairs of randomly sampled video frames and asymmetrically masks them. It outperforms state-of-the-art self-supervised methods on video object segmentation, pose keypoint propagation, and semantic part propagation tasks.
  arXiv  Detail & Related papers  (2023-05-23T17:59:46Z)
• Mask to reconstruct: Cooperative Semantics Completion for Video-text Retrieval [19.61947785487129]
  Mask for Semantics Completion (MASCOT) based on semantic-based masked modeling. Our MASCOT performs state-of-the-art performance on four major text-video retrieval benchmarks.
  arXiv  Detail & Related papers  (2023-05-13T12:31:37Z)
• EVEREST: Efficient Masked Video Autoencoder by Removing Redundant Spatiotemporal Tokens [57.354304637367555]
  We present EVEREST, a surprisingly efficient MVA approach for video representation learning. It finds tokens containing rich motion features and discards uninformative ones during both pre-training and fine-tuning. Our method significantly reduces the computation and memory requirements of MVA.
  arXiv  Detail & Related papers  (2022-11-19T09:57:01Z)
• Masked Motion Encoding for Self-Supervised Video Representation Learning [84.24773072241945]
  We present Masked Motion MME, a new pre-training paradigm that reconstructs both appearance and motion information to explore temporal clues. Motivated by the fact that human is able to recognize an action by tracking objects' position changes and shape changes, we propose to reconstruct a motion trajectory that represents these two kinds of change in the masked regions. Pre-trained with our MME paradigm, the model is able to anticipate long-term and fine-grained motion details.
  arXiv  Detail & Related papers  (2022-10-12T11:19:55Z)
• DeViT: Deformed Vision Transformers in Video Inpainting [59.73019717323264]
  We extend previous Transformers with patch alignment by introducing Deformed Patch-based Homography (DePtH) Second, we introduce Mask Pruning-based Patch Attention (MPPA) to improve patch-wised feature matching. Third, we introduce a Spatial-Temporal weighting Adaptor (STA) module to obtain accurate attention to spatial-temporal tokens.
  arXiv  Detail & Related papers  (2022-09-28T08:57:14Z)
• Video Demoireing with Relation-Based Temporal Consistency [68.20281109859998]
  Moire patterns, appearing as color distortions, severely degrade image and video qualities when filming a screen with digital cameras. We study how to remove such undesirable moire patterns in videos, namely video demoireing.
  arXiv  Detail & Related papers  (2022-04-06T17:45:38Z)
• VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling [88.30109041658618]
  A great challenge in video-language (VidL) modeling lies in the disconnection between fixed video representations extracted from image/video understanding models and downstream VidL data. We present VIOLET, a fully end-to-end VIdeO-LanguagE Transformer, which adopts a video transformer to explicitly model the temporal dynamics of video inputs.
  arXiv  Detail & Related papers  (2021-11-24T18:31:20Z)
• Occlusion-Aware Video Object Inpainting [72.38919601150175]
  This paper presents occlusion-aware video object inpainting, which recovers both the complete shape and appearance for occluded objects in videos. Our technical contribution VOIN jointly performs video object shape completion and occluded texture generation. For more realistic results, VOIN is optimized using both T-PatchGAN and a newoc-temporal YouTube attention-based multi-class discriminator.
  arXiv  Detail & Related papers  (2021-08-15T15:46:57Z)
• Learning Joint Spatial-Temporal Transformations for Video Inpainting [58.939131620135235]
  We propose to learn a joint Spatial-Temporal Transformer Network (STTN) for video inpainting. We simultaneously fill missing regions in all input frames by self-attention, and propose to optimize STTN by a spatial-temporal adversarial loss.
  arXiv  Detail & Related papers  (2020-07-20T16:35:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.