GS-VTON: Controllable 3D Virtual Try-on with Gaussian Splatting

• URL: http://arxiv.org/abs/2410.05259v1
• Date: Mon, 07 Oct 2024 17:58:20 GMT
• Title: GS-VTON: Controllable 3D Virtual Try-on with Gaussian Splatting
• Authors: Yukang Cao, Masoud Hadi, Liang Pan, Ziwei Liu,
• Abstract summary:
• Score: 0.0
• License:
• Abstract: Diffusion-based 2D virtual try-on (VTON) techniques have recently demonstrated strong performance, while the development of 3D VTON has largely lagged behind. Despite recent advances in text-guided 3D scene editing, integrating 2D VTON into these pipelines to achieve vivid 3D VTON remains challenging. The reasons are twofold. First, text prompts cannot provide sufficient details in describing clothing. Second, 2D VTON results generated from different viewpoints of the same 3D scene lack coherence and spatial relationships, hence frequently leading to appearance inconsistencies and geometric distortions. To resolve these problems, we introduce an image-prompted 3D VTON method (dubbed GS-VTON) which, by leveraging 3D Gaussian Splatting (3DGS) as the 3D representation, enables the transfer of pre-trained knowledge from 2D VTON models to 3D while improving cross-view consistency. (1) Specifically, we propose a personalized diffusion model that utilizes low-rank adaptation (LoRA) fine-tuning to incorporate personalized information into pre-trained 2D VTON models. To achieve effective LoRA training, we introduce a reference-driven image editing approach that enables the simultaneous editing of multi-view images while ensuring consistency. (2) Furthermore, we propose a persona-aware 3DGS editing framework to facilitate effective editing while maintaining consistent cross-view appearance and high-quality 3D geometry. (3) Additionally, we have established a new 3D VTON benchmark, 3D-VTONBench, which facilitates comprehensive qualitative and quantitative 3D VTON evaluations. Through extensive experiments and comparative analyses with existing methods, the proposed \OM has demonstrated superior fidelity and advanced editing capabilities, affirming its effectiveness for 3D VTON.

Related papers

• DINO in the Room: Leveraging 2D Foundation Models for 3D Segmentation [51.43837087865105]
  Vision foundation models (VFMs) trained on large-scale image datasets provide high-quality features that have significantly advanced 2D visual recognition.<n>Their potential in 3D vision remains largely untapped, despite the common availability of 2D images alongside 3D point cloud datasets.<n>We introduce DITR, a simple yet effective approach that extracts 2D foundation model features, projects them to 3D, and finally injects them into a 3D point cloud segmentation model.
  arXiv  Detail & Related papers  (2025-03-24T17:59:11Z)
• VTON 360: High-Fidelity Virtual Try-On from Any Viewing Direction [103.0918705283309]
  Virtual Try-On (VTON) is a transformative technology in e-commerce and fashion design, enabling realistic digital visualization of clothing on individuals.<n>We propose VTON 360, a novel 3D VTON method that addresses the open challenge of achieving high-fidelity VTON that supports any-view rendering.
  arXiv  Detail & Related papers  (2025-03-15T15:08:48Z)
• Unifying 2D and 3D Vision-Language Understanding [85.84054120018625]
  We introduce UniVLG, a unified architecture for 2D and 3D vision-language learning.<n>UniVLG bridges the gap between existing 2D-centric models and the rich 3D sensory data available in embodied systems.
  arXiv  Detail & Related papers  (2025-03-13T17:56:22Z)
• SeeGround: See and Ground for Zero-Shot Open-Vocabulary 3D Visual Grounding [10.81711535075112]
  3D Visual Grounding aims to locate objects in 3D scenes based on textual descriptions, essential for applications like augmented reality and robotics.<n>We introduce SeeGround, a zero-shot 3DVG framework leveraging 2D Vision-Language Models (VLMs) trained on large-scale 2D data.<n>SeeGround represents 3D scenes as a hybrid of query-aligned rendered images and spatially enriched text descriptions, bridging the gap between 3D data and 2D-VLMs input formats.
  arXiv  Detail & Related papers  (2024-12-05T17:58:43Z)
• DreamVTON: Customizing 3D Virtual Try-on with Personalized Diffusion Models [56.55549019625362]
  Image-based 3D Virtual Try-ON (VTON) aims to sculpt the 3D human according to person and clothes images. Recent text-to-3D methods achieve remarkable improvement in high-fidelity 3D human generation. We propose a novel customizing 3D human try-on model, named textbfDreamVTON, to separately optimize the geometry and texture of the 3D human.
  arXiv  Detail & Related papers  (2024-07-23T14:25:28Z)
• GaussianVTON: 3D Human Virtual Try-ON via Multi-Stage Gaussian Splatting Editing with Image Prompting [2.2975420753582028]
  E-commerce has underscored the importance of Virtual Try-On (VTON) Research on 3D VTON primarily centers on garment-body shape compatibility. Thanks to advances in 3D scene editing, a 2D diffusion model has now been adapted for 3D editing via multi-viewpoint editing.
  arXiv  Detail & Related papers  (2024-05-13T05:18:07Z)
• SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusion [33.69006364120861]
  We present Stable Video 3D (SV3D) -- a latent video diffusion model for high-resolution, image-to-multi-view generation of orbital videos around a 3D object.
  arXiv  Detail & Related papers  (2024-03-18T17:46:06Z)
• Weakly Supervised Monocular 3D Detection with a Single-View Image [58.57978772009438]
  Monocular 3D detection aims for precise 3D object localization from a single-view image. We propose SKD-WM3D, a weakly supervised monocular 3D detection framework. We show that SKD-WM3D surpasses the state-of-the-art clearly and is even on par with many fully supervised methods.
  arXiv  Detail & Related papers  (2024-02-29T13:26:47Z)
• 3D-to-2D Distillation for Indoor Scene Parsing [78.36781565047656]
  We present a new approach that enables us to leverage 3D features extracted from large-scale 3D data repository to enhance 2D features extracted from RGB images. First, we distill 3D knowledge from a pretrained 3D network to supervise a 2D network to learn simulated 3D features from 2D features during the training. Second, we design a two-stage dimension normalization scheme to calibrate the 2D and 3D features for better integration. Third, we design a semantic-aware adversarial training model to extend our framework for training with unpaired 3D data.
  arXiv  Detail & Related papers  (2021-04-06T02:22:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.