OmniDiT: Extending Diffusion Transformer to Omni-VTON Framework

• URL: http://arxiv.org/abs/2603.19643v2
• Date: Tue, 24 Mar 2026 03:23:30 GMT
• Title: OmniDiT: Extending Diffusion Transformer to Omni-VTON Framework
• Abstract summary: We propose OmniDiT, an omni Virtual Try-On framework based on the Diffusion Transformer.<n>We first establish a self-evolving data curation pipeline to continuously produce data, and construct a large VTON dataset Omni-TryOn.<n>We are the first to introduce Shifted Window Attention into the diffusion model, thus achieving a linear complexity.
• Score: 16.89102936125974
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Despite the rapid advancement of Virtual Try-On (VTON) and Try-Off (VTOFF) technologies, existing VTON methods face challenges with fine-grained detail preservation, generalization to complex scenes, complicated pipeline, and efficient inference. To tackle these problems, we propose OmniDiT, an omni Virtual Try-On framework based on the Diffusion Transformer, which combines try-on and try-off tasks into one unified model. Specifically, we first establish a self-evolving data curation pipeline to continuously produce data, and construct a large VTON dataset Omni-TryOn, which contains over 380k diverse and high-quality garment-model-tryon image pairs and detailed text prompts. Then, we employ the token concatenation and design an adaptive position encoding to effectively incorporate multiple reference conditions. To relieve the bottleneck of long sequence computation, we are the first to introduce Shifted Window Attention into the diffusion model, thus achieving a linear complexity. To remedy the performance degradation caused by local window attention, we utilize multiple timestep prediction and an alignment loss to improve generation fidelity. Experiments reveal that, under various complex scenes, our method achieves the best performance in both the model-free VTON and VTOFF tasks and a performance comparable to current SOTA methods in the model-based VTON task.

Related papers

• Structure-Detail Decoupled Autoregressive Generation for Fast and High-Fidelity Virtual Try-On [51.92060872944388]
  Virtual try-on (VTON) is a bi-conditional image generation problem.<n> Diffusion-based VTON methods suffer from high-frequency detail loss due to inherent latent compression.<n>Recent visual autoregressive models offer a promising alternative for high-quality generation.
  arXiv  Detail & Related papers  (2026-07-13T08:21:43Z)
• PROMO: Promptable Outfitting for Efficient High-Fidelity Virtual Try-On [39.47263059671598]
  VTON is a structured image editing problem that demands strong conditional generation under three key requirements.<n>We present PROMO, a promptable virtual try-on framework built upon a Flow Matching DiT backbone with latent multi-modal conditional concatenation.<n>On standard benchmarks, PROMO surpasses both prior VTON methods and general image editing models in visual fidelity while delivering a competitive balance between quality and speed.
  arXiv  Detail & Related papers  (2026-03-12T08:41:05Z)
• Align-Then-stEer: Adapting the Vision-Language Action Models through Unified Latent Guidance [63.33213516925946]
  We introduce textbfAlign-Then-stEer (textttATE), a novel, data-efficient, and plug-and-play adaptation framework.<n>Our work presents a general and lightweight solution that greatly enhances the practicality of deploying VLA models to new robotic platforms and tasks.
  arXiv  Detail & Related papers  (2025-09-02T07:51:59Z)
• ITVTON: Virtual Try-On Diffusion Transformer Based on Integrated Image and Text [1.7071356210178177]
  ITVTON is an efficient framework that leverages the Diffusion Transformer (DiT) as its single generator to improve image fidelity.<n>ITVTON effectively captures garment and person images along the width dimension and incorporating textual descriptions from both.<n>Experiments on 10,257 image pairs from IGPair confirm ITVTON's robustness in real-world scenarios.
  arXiv  Detail & Related papers  (2025-01-28T07:24:15Z)
• 1-2-1: Renaissance of Single-Network Paradigm for Virtual Try-On [17.226542332700607]
  We propose a novel single-network VTON method that overcomes the limitations of existing techniques.<n>Our method, namely MNVTON, introduces a Modality-specific Normalization strategy that separately processes text, image and video inputs.<n>Our results suggest that the single-network paradigm can rival the performance of dualnetwork approaches.
  arXiv  Detail & Related papers  (2025-01-09T16:49:04Z)
• LaVin-DiT: Large Vision Diffusion Transformer [99.98106406059333]
  LaVin-DiT is a scalable and unified foundation model designed to tackle over 20 computer vision tasks in a generative framework.<n>We introduce key innovations to optimize generative performance for vision tasks.<n>The model is scaled from 0.1B to 3.4B parameters, demonstrating substantial scalability and state-of-the-art performance across diverse vision tasks.
  arXiv  Detail & Related papers  (2024-11-18T12:05:27Z)
• Visual Fourier Prompt Tuning [63.66866445034855]
  We propose the Visual Fourier Prompt Tuning (VFPT) method as a general and effective solution for adapting large-scale transformer-based models. Our approach incorporates the Fast Fourier Transform into prompt embeddings and harmoniously considers both spatial and frequency domain information. Our results demonstrate that our approach outperforms current state-of-the-art baselines on two benchmarks.
  arXiv  Detail & Related papers  (2024-11-02T18:18:35Z)
• VST++: Efficient and Stronger Visual Saliency Transformer [74.26078624363274]
  We develop an efficient and stronger VST++ model to explore global long-range dependencies. We evaluate our model across various transformer-based backbones on RGB, RGB-D, and RGB-T SOD benchmark datasets.
  arXiv  Detail & Related papers  (2023-10-18T05:44:49Z)
• An Empirical Study of Training End-to-End Vision-and-Language Transformers [50.23532518166621]
  We present METER(textbfMultimodal textbfEnd-to-end textbfTransformtextbfER), through which we investigate how to design and pre-train a fully transformer-based VL model. Specifically, we dissect the model designs along multiple dimensions: vision encoders (e.g., CLIP-ViT, Swin transformer), text encoders (e.g., RoBERTa, DeBERTa), multimodal fusion (e.g., merged attention vs. co-
  arXiv  Detail & Related papers  (2021-11-03T17:55:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.