InstructVVT: Instruction-Driven Video Virtual Try-On without Auxiliary Spatial Priors

• URL: http://arxiv.org/abs/2608.14070v1
• Date: Fri, 14 Aug 2026 08:26:42 GMT
• Title: InstructVVT: Instruction-Driven Video Virtual Try-On without Auxiliary Spatial Priors
• Abstract summary: InstructVVT is an instruction-driven and reference-guided video virtual try-on framework.<n>We show that InstructVVT outperforms state-of-the-art open-source methods in garment fidelity, structural preservation, and temporal consistency.
• Score: 47.314679133060274
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Video virtual try-on is a highly constrained editing task requiring the precise replacement of a target person's clothing while strictly preserving the original video's spatial structure and temporal dynamics. Existing methods heavily rely on auxiliary handcrafted spatial priors (e.g., masks, poses) for editing control. However, these priors are prone to failure in unconstrained real-world videos and often compress rich visual context into incomplete structural signals. Furthermore, standard reconstruction objectives fail to fully capture try-on-specific human preferences. To address these challenges, we propose InstructVVT, an instruction-driven and reference-guided video virtual try-on framework based on a Diffusion Transformer (DiT) that operates without inference-time spatial priors. Our core insight is to recover fine-grained control directly from the input triplet (source video, reference garment, and instruction) via a dual-level reference conditioning scheme. Specifically, an MLLM infers semantic edit tokens for target disambiguation and structural preservation, while a lightweight conditioning pathway explicitly injects fine-grained visual garment details. Finally, we design a try-on-specific reward and utilize the DiffusionNFT algorithm to align the model with human preferences. Extensive experiments on ViViD-S and TripVVT-Bench demonstrate that InstructVVT outperforms state-of-the-art open-source methods in garment fidelity, structural preservation, and temporal consistency, despite requiring fewer inference-time controls.

Related papers

• UniVVT: A Unified End-to-End Framework for High-Fidelity Video Virtual Try-on [21.69586852578892]
  Video Virtual Try-On (VVT) synthesizes a video of a person wearing a target garment while preserving identity, motion, and scene dynamics.<n>Traditional approaches cast VVT as mask-conditioned video inpainting and rely on separate modules for human parsing, pose estimation, and garment warping.<n>We present UniVVT, a unified end-to-end framework that reframes VVT as semantically conditioned video generation.
  arXiv  Detail & Related papers  (2026-08-06T08:29:47Z)
• STORM: Internalized Modeling for Spatial-Temporal Reasoning in Video-Language Models [81.32710031596591]
  Video reasoning tasks require tracking motion, temporal order, and evolving visual states across frames.<n>Existing methods built on vision-language models (LVLMs) often address this challenge by externalizing reasoning through chain-of-thought (CoT)<n>We propose STORMS, a two-stage framework that teaches LVLMs to reason through bounded continuous latent trajectories instead of explicit textual CoT.
  arXiv  Detail & Related papers  (2026-05-25T16:33:00Z)
• Compositional Video Generation via Inference-Time Guidance [69.53614395025632]
  Text-to-video diffusion models often fail on prompts requiring compositional understanding.<n>We propose textbfCVG, an inference-time guidance method for improving compositional faithfulness in frozen text-to-video models.
  arXiv  Detail & Related papers  (2026-05-14T15:50:25Z)
• VIRST: Video-Instructed Reasoning Assistant for SpatioTemporal Segmentation [6.447274127678917]
  Video Video Object (RVOS) aims to segment target objects in videos based on natural language descriptions.<n>We propose VIRST (Video-Instructed Reasoning Assistant for Spatio-Temporal), an end-to-end framework that unifies global video reasoning and pixel-level mask prediction within a single model.
  arXiv  Detail & Related papers  (2026-03-28T00:34:15Z)
• SAMA: Factorized Semantic Anchoring and Motion Alignment for Instruction-Guided Video Editing [76.349958946335]
  We present SAMA (factorized Semantic Anchoring and Motion Alignment), a framework that factorizes video editing into semantic anchoring and motion modeling.<n>First, we introduce Semantic Anchoring, which establishes a reliable visual anchor by jointly predicting semantic tokens and video latents at sparse anchor frames.<n>Second, Motion Alignment pre-trains the same backbone on motion-centric video restoration pretext tasks.
  arXiv  Detail & Related papers  (2026-03-19T17:59:51Z)
• PVI: Plug-in Visual Injection for Vision-Language-Action Models [8.641415997612066]
  VLAs that pair a pretrained VLM with a flow-matching action expert have emerged as a strong paradigm for language-conditioned manipulation.<n>We propose Plug-in Visual Injection (PVI), a lightweight encoder-agnostic module that attaches to a pretrained action expert and injects auxiliary visual representations via zero-d residual pathways.
  arXiv  Detail & Related papers  (2026-03-13T08:16:53Z)
• PropFly: Learning to Propagate via On-the-Fly Supervision from Pre-trained Video Diffusion Models [35.59605874012795]
  PropFly is a training pipeline for propagation-based video editing.<n>PropFly relies on pre-trained video diffusion models (VDMs) instead of requiring off-the-shelf or precomputed paired video editing datasets.<n>Our pipeline enables an adapter attached to the pre-trained VDM to learn to propagate edits via Guidance-Modulated Flow Matching (GMFM) loss.
  arXiv  Detail & Related papers  (2026-02-24T06:11:08Z)
• A Reason-then-Describe Instruction Interpreter for Controllable Video Generation [88.95178842901095]
  We propose ReaDe, a universal, model-agnostic interpreter that converts raw instructions into precise, actionable specifications for downstream video generators.<n>We show consistent gains in instruction fidelity, caption accuracy, and downstream video quality, with strong generalization to reasoning-intensive and unseen inputs.
  arXiv  Detail & Related papers  (2025-11-25T17:59:07Z)
• PEMF-VTO: Point-Enhanced Video Virtual Try-on via Mask-free Paradigm [21.1235226974745]
  Video Virtual Try-on aims to seamlessly transfer a reference garment onto a target person in a video.<n>Existing methods typically rely on inpainting masks to define the try-on area.<n>We propose PEMF-VTO, a novel Point-Enhanced Mask-Free Video Virtual Try-On framework.
  arXiv  Detail & Related papers  (2024-12-04T04:24:15Z)
• Betrayed by Attention: A Simple yet Effective Approach for Self-supervised Video Object Segmentation [76.68301884987348]
  We propose a simple yet effective approach for self-supervised video object segmentation (VOS) Our key insight is that the inherent structural dependencies present in DINO-pretrained Transformers can be leveraged to establish robust-temporal segmentation correspondences in videos. Our method demonstrates state-of-the-art performance across multiple unsupervised VOS benchmarks and excels in complex real-world multi-object video segmentation tasks.
  arXiv  Detail & Related papers  (2023-11-29T18:47:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.