Scalable Motion Style Transfer with Constrained Diffusion Generation

• URL: http://arxiv.org/abs/2312.07311v1
• Date: Tue, 12 Dec 2023 14:28:31 GMT
• Title: Scalable Motion Style Transfer with Constrained Diffusion Generation
• Abstract summary: Current training of style transfer systems relies on consistency losses across style domains to preserve contents. Recent image transfer works show the potential of independent training on each domain by leveraging implicit bridging between diffusion models. We address this by imposing biased sampling in backward diffusion while maintaining the domain independence in the training stage.
• Score: 27.330473515686187
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Current training of motion style transfer systems relies on consistency losses across style domains to preserve contents, hindering its scalable application to a large number of domains and private data. Recent image transfer works show the potential of independent training on each domain by leveraging implicit bridging between diffusion models, with the content preservation, however, limited to simple data patterns. We address this by imposing biased sampling in backward diffusion while maintaining the domain independence in the training stage. We construct the bias from the source domain keyframes and apply them as the gradient of content constraints, yielding a framework with keyframe manifold constraint gradients (KMCGs). Our validation demonstrates the success of training separate models to transfer between as many as ten dance motion styles. Comprehensive experiments find a significant improvement in preserving motion contents in comparison to baseline and ablative diffusion-based style transfer models. In addition, we perform a human study for a subjective assessment of the quality of generated dance motions. The results validate the competitiveness of KMCGs.

Related papers

• EvBS: Event-guided Blur Synthesis for Domain-adaptive Motion Deblurring [22.317727241744702]
  EvBS is an event-guided blur synthesis framework.<n>It generates diverse training pairs for calibrating pre-trained models to the target domain.<n>EvBS effectively enhances the robustness of existing deblurring models on unseen testing datasets.
  arXiv  Detail & Related papers  (2026-08-08T11:10:26Z)
• MoASE++: Mixture of Activation Sparsity Experts with Domain-Adaptive On-policy Distillation for Continual Test Time Adaptation [65.068801413044]
  Continual test-time adaptation adapts a source-pretrained model to non-stationary, unlabeled target streams.<n>We introduce MoASE, a mixture-of-experts that disentangles domain-agnostic structure from domain-specific texture.
  arXiv  Detail & Related papers  (2026-05-18T01:52:12Z)
• Style Transfer with Diffusion Models for Synthetic-to-Real Domain Adaptation [4.50001192781448]
  We introduce two novel techniques for semantically consistent style transfer using diffusion models.<n>Experiments using GTA5 as source and Cityscapes/ACDC as target domains show that our approach produces higher quality images with lower FID scores and better content preservation.
  arXiv  Detail & Related papers  (2025-05-22T08:11:10Z)
• Improving Vector-Quantized Image Modeling with Latent Consistency-Matching Diffusion [55.185588994883226]
  We introduce VQ-LCMD, a continuous-space latent diffusion framework within the embedding space that stabilizes training. VQ-LCMD uses a novel training objective combining the joint embedding-diffusion variational lower bound with a consistency-matching (CM) loss. Experiments show that the proposed VQ-LCMD yields superior results on FFHQ, LSUN Churches, and LSUN Bedrooms compared to discrete-state latent diffusion models.
  arXiv  Detail & Related papers  (2024-10-18T09:12:33Z)
• Denoising as Adaptation: Noise-Space Domain Adaptation for Image Restoration [64.84134880709625]
  We show that it is possible to perform domain adaptation via the noise space using diffusion models. In particular, by leveraging the unique property of how auxiliary conditional inputs influence the multi-step denoising process, we derive a meaningful diffusion loss. We present crucial strategies such as channel-shuffling layer and residual-swapping contrastive learning in the diffusion model.
  arXiv  Detail & Related papers  (2024-06-26T17:40:30Z)
• StyDeSty: Min-Max Stylization and Destylization for Single Domain Generalization [85.18995948334592]
  Single domain generalization (single DG) aims at learning a robust model generalizable to unseen domains from only one training domain. State-of-the-art approaches have mostly relied on data augmentations, such as adversarial perturbation and style enhancement, to synthesize new data. We propose emphStyDeSty, which explicitly accounts for the alignment of the source and pseudo domains in the process of data augmentation.
  arXiv  Detail & Related papers  (2024-06-01T02:41:34Z)
• Diffusion-based Human Motion Style Transfer with Semantic Guidance [23.600154466988073]
  We propose a novel framework for few-shot style transfer learning based on the diffusion model. In the first stage, we pre-train a diffusion-based text-to-motion model as a generative prior. In the second stage, based on the single style example, we fine-tune the pre-trained diffusion model in a few-shot manner to make it capable of style transfer.
  arXiv  Detail & Related papers  (2024-03-20T05:52:11Z)
• Expanding Expressiveness of Diffusion Models with Limited Data via Self-Distillation based Fine-Tuning [24.791783885165923]
  Training diffusion models on limited datasets poses challenges in terms of limited generation capacity and expressiveness. We propose Self-Distillation for Fine-Tuning diffusion models (SDFT) to address these challenges.
  arXiv  Detail & Related papers  (2023-11-02T06:24:06Z)
• Phasic Content Fusing Diffusion Model with Directional Distribution Consistency for Few-Shot Model Adaption [73.98706049140098]
  We propose a novel phasic content fusing few-shot diffusion model with directional distribution consistency loss. Specifically, we design a phasic training strategy with phasic content fusion to help our model learn content and style information when t is large. Finally, we propose a cross-domain structure guidance strategy that enhances structure consistency during domain adaptation.
  arXiv  Detail & Related papers  (2023-09-07T14:14:11Z)
• A Novel Cross-Perturbation for Single Domain Generalization [54.612933105967606]
  Single domain generalization aims to enhance the ability of the model to generalize to unknown domains when trained on a single source domain. The limited diversity in the training data hampers the learning of domain-invariant features, resulting in compromised generalization performance. We propose CPerb, a simple yet effective cross-perturbation method to enhance the diversity of the training data.
  arXiv  Detail & Related papers  (2023-08-02T03:16:12Z)
• Continual Vision-Language Representation Learning with Off-Diagonal Information [112.39419069447902]
  Multi-modal contrastive learning frameworks like CLIP typically require a large amount of image-text samples for training. This paper discusses the feasibility of continual CLIP training using streaming data.
  arXiv  Detail & Related papers  (2023-05-11T08:04:46Z)
• Variational Transfer Learning using Cross-Domain Latent Modulation [1.9662978733004601]
  We introduce a novel cross-domain latent modulation mechanism to a variational autoencoder framework so as to achieve effective transfer learning. Deep representations of the source and target domains are first extracted by a unified inference model and aligned by employing gradient reversal. The learned deep representations are then cross-modulated to the latent encoding of the alternative domain, where consistency constraints are also applied.
  arXiv  Detail & Related papers  (2022-05-31T03:47:08Z)
• Consistency and Diversity induced Human Motion Segmentation [231.36289425663702]
  We propose a novel Consistency and Diversity induced human Motion (CDMS) algorithm. Our model factorizes the source and target data into distinct multi-layer feature spaces. A multi-mutual learning strategy is carried out to reduce the domain gap between the source and target data.
  arXiv  Detail & Related papers  (2022-02-10T06:23:56Z)
• Cross-Domain Latent Modulation for Variational Transfer Learning [1.9212368803706577]
  We propose a cross-domain latent modulation mechanism within a variational autoencoders (VAE) framework to enable improved transfer learning. We apply the proposed model to a number of transfer learning tasks including unsupervised domain adaptation and image-to-image translation.
  arXiv  Detail & Related papers  (2020-12-21T22:45:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.