SoPo: Text-to-Motion Generation Using Semi-Online Preference Optimization

• URL: http://arxiv.org/abs/2412.05095v2
• Date: Wed, 28 May 2025 05:23:44 GMT
• Title: SoPo: Text-to-Motion Generation Using Semi-Online Preference Optimization
• Authors: Xiaofeng Tan, Hongsong Wang, Xin Geng, Pan Zhou,
• Abstract summary: We focus on fine-tuning text-to-motion models to consistently favor high-quality, human-preferred motions.<n>We introduce Semi-online Preference Optimization (SoPo), a DPO-based method for training text-to-motion models.<n>Experiments demonstrate that SoPo outperforms other preference alignment methods.
• Score: 82.83603957387442
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Text-to-motion generation is essential for advancing the creative industry but often presents challenges in producing consistent, realistic motions. To address this, we focus on fine-tuning text-to-motion models to consistently favor high-quality, human-preferred motions, a critical yet largely unexplored problem. In this work, we theoretically investigate the DPO under both online and offline settings, and reveal their respective limitation: overfitting in offline DPO, and biased sampling in online DPO. Building on our theoretical insights, we introduce Semi-online Preference Optimization (SoPo), a DPO-based method for training text-to-motion models using "semi-online" data pair, consisting of unpreferred motion from online distribution and preferred motion in offline datasets. This method leverages both online and offline DPO, allowing each to compensate for the other's limitations. Extensive experiments demonstrate that SoPo outperforms other preference alignment methods, with an MM-Dist of 3.25% (vs e.g. 0.76% of MoDiPO) on the MLD model, 2.91% (vs e.g. 0.66% of MoDiPO) on MDM model, respectively. Additionally, the MLD model fine-tuned by our SoPo surpasses the SoTA model in terms of R-precision and MM Dist. Visualization results also show the efficacy of our SoPo in preference alignment. Project page: https://xiaofeng-tan.github.io/projects/SoPo/ .

Related papers

• Aligning Generative Speech Enhancement with Human Preferences via Direct Preference Optimization [46.94426003410216]
  This work investigates speech enhancement from the perspective of language models (LMs)<n>Using UTMOS, a neural MOS prediction model, as a proxy for human ratings, our approach guides optimization toward perceptually preferred outputs.<n>Experiments on the 2020 Deep Noise Suppression Challenge test sets demonstrate that applying DPO to a pretrained LM-based SE model yields consistent improvements.
  arXiv  Detail & Related papers  (2025-07-14T05:15:39Z)
• Self-NPO: Negative Preference Optimization of Diffusion Models by Simply Learning from Itself without Explicit Preference Annotations [60.143658714894336]
  Diffusion models have demonstrated remarkable success in various visual generation tasks, including image, video, and 3D content generation.<n> Preference optimization (PO) is a prominent and growing area of research that aims to align these models with human preferences.<n>We introduce Self-NPO, a Negative Preference Optimization approach that learns exclusively from the model itself.
  arXiv  Detail & Related papers  (2025-05-17T01:03:46Z)
• InPO: Inversion Preference Optimization with Reparametrized DDIM for Efficient Diffusion Model Alignment [12.823734370183482]
  We introduce DDIM-InPO, an efficient method for direct preference alignment of diffusion models. Our approach conceptualizes diffusion model as a single-step generative model, allowing us to fine-tune the outputs of specific latent variables selectively. Experimental results indicate that our DDIM-InPO achieves state-of-the-art performance with just 400 steps of fine-tuning.
  arXiv  Detail & Related papers  (2025-03-24T08:58:49Z)
• CHiP: Cross-modal Hierarchical Direct Preference Optimization for Multimodal LLMs [107.21334626890713]
  Multimodal Large Language Models (MLLMs) still struggle with hallucinations despite their impressive capabilities. We propose a Cross-modal Hierarchical Direct Preference Optimization (CHiP) to address these limitations. We evaluate CHiP through both quantitative and qualitative analyses, with results across multiple benchmarks demonstrating its effectiveness in reducing hallucinations.
  arXiv  Detail & Related papers  (2025-01-28T02:05:38Z)
• Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimization [75.1240295759264]
  We propose an effective framework for Bridging and Modeling Correlations in pairwise data, named BMC. We increase the consistency and informativeness of the pairwise preference signals through targeted modifications. We identify that DPO alone is insufficient to model these correlations and capture nuanced variations.
  arXiv  Detail & Related papers  (2024-08-14T11:29:47Z)
• Self-Augmented Preference Optimization: Off-Policy Paradigms for Language Model Alignment [104.18002641195442]
  We introduce Self-Augmented Preference Optimization (SAPO), an effective and scalable training paradigm that does not require existing paired data. Building on the self-play concept, which autonomously generates negative responses, we further incorporate an off-policy learning pipeline to enhance data exploration and exploitation.
  arXiv  Detail & Related papers  (2024-05-31T14:21:04Z)
• MoDiPO: text-to-motion alignment via AI-feedback-driven Direct Preference Optimization [6.147750347011554]
  We propose MoDiPO (Motion Diffusion DPO) to align text-to-motion models. We streamline the laborious and expensive process of gathering human preferences needed in DPO by leveraging AI feedback. We demonstrate, both qualitatively and quantitatively, that our proposed method yields significantly more realistic motions.
  arXiv  Detail & Related papers  (2024-05-06T19:19:20Z)
• Self-Play Preference Optimization for Language Model Alignment [75.83359213697854]
  Recent advancements suggest that directly working with preference probabilities can yield a more accurate reflection of human preferences. We propose a self-play-based method for language model alignment, which treats the problem as a constant-sum two-player game. Our approach, dubbed Self-Play Preference Optimization (SPPO), utilizes iterative policy updates to provably approximate the Nash equilibrium.
  arXiv  Detail & Related papers  (2024-05-01T17:59:20Z)
• Weak-to-Strong Extrapolation Expedites Alignment [135.12769233630362]
  We propose a method called ExPO to boost models' alignment with human preference. We demonstrate that ExPO consistently improves off-the-shelf DPO/RLHF models. We shed light on the essence of ExPO amplifying the reward signal learned during alignment training.
  arXiv  Detail & Related papers  (2024-04-25T17:39:50Z)
• Human Alignment of Large Language Models through Online Preference Optimisation [50.52545798589968]
  We show the equivalence between two recent alignment methods, namely Identity Policy optimisation (IPO) and Nash Mirror Descent (Nash-MD) This equivalence can be proven when we consider the online version of IPO, that is when both generations are sampled by the online policy and annotated by a trained preference model. We introduce the IPO-MD algorithm that generates data with a mixture policy (between the online and reference policy) similarly as the general Nash-MD algorithm.
  arXiv  Detail & Related papers  (2024-03-13T15:47:26Z)
• Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive [15.066029556877721]
  We show theoretically that the standard DPO loss can lead to a reduction of the model's likelihood of the preferred examples. We design DPO-Positive (DPOP), a new loss function and training procedure which avoids this failure mode. Surprisingly, we find that DPOP outperforms DPO and other fine-tuning procedures across a wide variety of datasets and downstream tasks.
  arXiv  Detail & Related papers  (2024-02-20T18:42:34Z)
• Self-Play Fine-Tuning of Diffusion Models for Text-to-Image Generation [59.184980778643464]
  Fine-tuning Diffusion Models remains an underexplored frontier in generative artificial intelligence (GenAI) In this paper, we introduce an innovative technique called self-play fine-tuning for diffusion models (SPIN-Diffusion) Our approach offers an alternative to conventional supervised fine-tuning and RL strategies, significantly improving both model performance and alignment.
  arXiv  Detail & Related papers  (2024-02-15T18:59:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.