ProFashion: Prototype-guided Fashion Video Generation with Multiple Reference Images

• URL: http://arxiv.org/abs/2505.06537v1
• Date: Sat, 10 May 2025 06:59:24 GMT
• Title: ProFashion: Prototype-guided Fashion Video Generation with Multiple Reference Images
• Authors: Xianghao Kong, Qiaosong Qi, Yuanbin Wang, Anyi Rao, Biaolong Chen, Aixi Zhang, Si Liu, Hao Jiang,
• Abstract summary: Fashion generation aims to synthesize temporally consistent videos from reference images of designated character.<n>We propose ProFashion, a framework to achieve improved consistency and temporal coherency.<n>We evaluate our method on the MRFashion-7K dataset we collected from the Internet.
• Score: 15.345172917297246
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Fashion video generation aims to synthesize temporally consistent videos from reference images of a designated character. Despite significant progress, existing diffusion-based methods only support a single reference image as input, severely limiting their capability to generate view-consistent fashion videos, especially when there are different patterns on the clothes from different perspectives. Moreover, the widely adopted motion module does not sufficiently model human body movement, leading to sub-optimal spatiotemporal consistency. To address these issues, we propose ProFashion, a fashion video generation framework leveraging multiple reference images to achieve improved view consistency and temporal coherency. To effectively leverage features from multiple reference images while maintaining a reasonable computational cost, we devise a Pose-aware Prototype Aggregator, which selects and aggregates global and fine-grained reference features according to pose information to form frame-wise prototypes, which serve as guidance in the denoising process. To further enhance motion consistency, we introduce a Flow-enhanced Prototype Instantiator, which exploits the human keypoint motion flow to guide an extra spatiotemporal attention process in the denoiser. To demonstrate the effectiveness of ProFashion, we extensively evaluate our method on the MRFashion-7K dataset we collected from the Internet. ProFashion also outperforms previous methods on the UBC Fashion dataset.

Related papers

• Multi-subject Open-set Personalization in Video Generation [110.02124633005516]
  We present Video Alchemist $-$ a video model with built-in multi-subject, open-set personalization capabilities.<n>Our model is built on a new Diffusion Transformer module that fuses each conditional reference image and its corresponding subject-level text prompt.<n>Our method significantly outperforms existing personalization methods in both quantitative and qualitative evaluations.
  arXiv  Detail & Related papers  (2025-01-10T18:59:54Z)
• MoTrans: Customized Motion Transfer with Text-driven Video Diffusion Models [59.10171699717122]
  MoTrans is a customized motion transfer method enabling video generation of similar motion in new context.<n> multimodal representations from recaptioned prompt and video frames promote the modeling of appearance.<n>Our method effectively learns specific motion pattern from singular or multiple reference videos.
  arXiv  Detail & Related papers  (2024-12-02T10:07:59Z)
• Pose-Guided Fine-Grained Sign Language Video Generation [18.167413937989867]
  We propose a novel Pose-Guided Motion Model (PGMM) for generating fine-grained and motion-consistent sign language videos. Firstly, we propose a new Coarse Motion Module (CMM), which completes the deformation of features by optical flow warping. Secondly, we propose a new Pose Fusion Module (PFM), which guides the modal fusion of RGB and pose features.
  arXiv  Detail & Related papers  (2024-09-25T07:54:53Z)
• VividPose: Advancing Stable Video Diffusion for Realistic Human Image Animation [79.99551055245071]
  We propose VividPose, an end-to-end pipeline that ensures superior temporal stability. An identity-aware appearance controller integrates additional facial information without compromising other appearance details. A geometry-aware pose controller utilizes both dense rendering maps from SMPL-X and sparse skeleton maps. VividPose exhibits superior generalization capabilities on our proposed in-the-wild dataset.
  arXiv  Detail & Related papers  (2024-05-28T13:18:32Z)
• RefDrop: Controllable Consistency in Image or Video Generation via Reference Feature Guidance [22.326405355520176]
  RefDrop allows users to control the influence of reference context in a direct and precise manner. Our method also enables more interesting applications, such as the consistent generation of multiple subjects.
  arXiv  Detail & Related papers  (2024-05-27T21:23:20Z)
• DreamPose: Fashion Image-to-Video Synthesis via Stable Diffusion [63.179505586264014]
  We present DreamPose, a diffusion-based method for generating animated fashion videos from still images. Given an image and a sequence of human body poses, our method synthesizes a video containing both human and fabric motion.
  arXiv  Detail & Related papers  (2023-04-12T17:59:17Z)
• DeepMultiCap: Performance Capture of Multiple Characters Using Sparse Multiview Cameras [63.186486240525554]
  DeepMultiCap is a novel method for multi-person performance capture using sparse multi-view cameras. Our method can capture time varying surface details without the need of using pre-scanned template models.
  arXiv  Detail & Related papers  (2021-05-01T14:32:13Z)
• Learning Long-Term Style-Preserving Blind Video Temporal Consistency [6.6908747077585105]
  We propose a postprocessing model, to the transformation applied to videos, in the form of a recurrent neural network. Our model is trained using a Ping Pong procedure and its corresponding loss, recently introduced for GAN video generation. We evaluate our model on the DAVIS and videvo.net datasets and show that our approach offers state-of-the-art results concerning flicker removal.
  arXiv  Detail & Related papers  (2021-03-12T13:54:34Z)
• ARVo: Learning All-Range Volumetric Correspondence for Video Deblurring [92.40655035360729]
  Video deblurring models exploit consecutive frames to remove blurs from camera shakes and object motions. We propose a novel implicit method to learn spatial correspondence among blurry frames in the feature space. Our proposed method is evaluated on the widely-adopted DVD dataset, along with a newly collected High-Frame-Rate (1000 fps) dataset for Video Deblurring.
  arXiv  Detail & Related papers  (2021-03-07T04:33:13Z)
• Adaptive Compact Attention For Few-shot Video-to-video Translation [13.535988102579918]
  We introduce a novel adaptive compact attention mechanism to efficiently extract contextual features jointly from multiple reference images. Our core idea is to extract compact basis sets from all the reference images as higher-level representations. We extensively evaluate our method on a large-scale talking-head video dataset and a human dancing dataset.
  arXiv  Detail & Related papers  (2020-11-30T11:19:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.