TPMM-DPO: Trajectory-aware Preference-guided Model Merging for Iterative Direct Preference Optimization

• URL: http://arxiv.org/abs/2605.23398v1
• Date: Fri, 22 May 2026 09:11:20 GMT
• Title: TPMM-DPO: Trajectory-aware Preference-guided Model Merging for Iterative Direct Preference Optimization
• Abstract summary: We propose TPMM-DPO, a trajectory-aware preference-guided model merging method.<n>In contrast to conventional iterative DPO, TPMM-DPO effectively mitigates error accumulation induced by noisy preferences.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Direct Preference Optimization (DPO) has been widely adopted for large language model alignment due to its simple training procedure and lack of an explicit reward model. However, in iterative DPO, when the policy model from the previous iteration is repeatedly used as the reference model for subsequent rounds, noise in preference data and errors in the reference model accumulate over time. This accumulation can lead to late-stage over-optimization, performance fluctuations, and degraded generalization. To address these issues, we propose TPMM-DPO, a trajectory-aware preference-guided model merging method. The method treats the sequence of policy models generated during iterative DPO as an optimization trajectory and adaptively integrates them using learned fusion weights, thereby constructing a smoother and more robust reference model. In contrast to conventional iterative DPO, which relies solely on a single previous model, TPMM-DPO effectively mitigates error accumulation induced by noisy preferences and improves training stability. Experimental results show that standard iterative DPO often suffers from performance degradation in the middle and later stages of training, whereas TPMM-DPO consistently improves generation quality and achieves higher win rates and reward scores on both in-domain and out-of-domain evaluations. Further ablation studies and robustness analyses demonstrate that, compared with simple averaging, learnable-weight fusion more effectively alleviates late-stage performance degradation caused by noisy preferences.

Related papers

• Linear-DPO: Linear Direct Preference Optimization for Diffusion and Flow-Matching Generative Models [5.527466367347639]
  We derive a generalized DPO objective that covers both diffusion and flow-matching via a unified reverse-time SDE framework.<n>We propose Linear-DPO, which replaces the aggressive sigmoid-based utility function with a sustained linear utility and incorporates an EMA-updated reference model.
  arXiv  Detail & Related papers  (2026-05-20T12:54:51Z)
• Gradient-Gated DPO: Stabilizing Preference Optimization in Language Models [0.0]
  We introduce Gradient-Gated Preference Optimization (Gate-DPO)<n>Gate-DPO stabilizes training by modulating rejected gradients according to the model's probability geometry.<n>Experiments across multiple architectures and preference datasets show that Gate-DPO consistently reduces squeezing and improves chosen-response likelihood.
  arXiv  Detail & Related papers  (2026-05-04T14:15:24Z)
• Unbiased Dynamic Pruning for Efficient Group-Based Policy Optimization [60.87651283510059]
  Group Relative Policy Optimization (GRPO) effectively scales LLM reasoning but incurs prohibitive computational costs.<n>We propose Dynamic Pruning Policy Optimization (DPPO), a framework that enables dynamic pruning while preserving unbiased gradient estimation.<n>To mitigate the data sparsity induced by pruning, we introduce Dense Prompt Packing, a window-based greedy strategy.
  arXiv  Detail & Related papers  (2026-03-04T14:48:53Z)
• Beyond Reward Margin: Rethinking and Resolving Likelihood Displacement in Diffusion Models via Video Generation [6.597818816347323]
  Direct Preference Optimization aims to align generative outputs with human preferences by distinguishing between chosen and rejected samples.<n>A critical limitation of DPO is likelihood displacement, where the probabilities of chosen samples paradoxically decrease during training.<n>We introduce a novel solution named Policy-Guided DPO, combining Adaptive Rejection Scaling (ARS) and Implicit Preference Regularization (IPR)<n>Experiments show that PG-DPO outperforms existing methods in both quantitative metrics and qualitative evaluations.
  arXiv  Detail & Related papers  (2025-11-24T12:37:49Z)
• Margin Adaptive DPO: Leveraging Reward Model for Granular Control in Preference Optimization [0.0]
  Margin-Adaptive Direct Preference Optimization provides a stable, data-preserving, and instance-level solution.<n>We provide a comprehensive theoretical analysis, proving that MADPO has a well-behaved optimization landscape.<n>It achieves performance gains of up to +33.3% on High Quality data and +10.5% on Low Quality data over the next-best method.
  arXiv  Detail & Related papers  (2025-10-06T20:09:37Z)
• Divergence Minimization Preference Optimization for Diffusion Model Alignment [66.31417479052774]
  Divergence Minimization Preference Optimization (DMPO) is a principled method for aligning diffusion models by minimizing reverse KL divergence.<n>DMPO can consistently outperform or match existing techniques across different base models and test sets.
  arXiv  Detail & Related papers  (2025-07-10T07:57:30Z)
• Self-Boost via Optimal Retraining: An Analysis via Approximate Message Passing [58.52119063742121]
  Retraining a model using its own predictions together with the original, potentially noisy labels is a well-known strategy for improving the model performance.<n>This paper addresses the question of how to optimally combine the model's predictions and the provided labels.<n>Our main contribution is the derivation of the Bayes optimal aggregator function to combine the current model's predictions and the given labels.
  arXiv  Detail & Related papers  (2025-05-21T07:16:44Z)
• Pre-DPO: Improving Data Utilization in Direct Preference Optimization Using a Guiding Reference Model [20.623037493149507]
  We propose Pre-DPO, a simple yet effective DPO-based training paradigm that enhances preference optimization performance by leveraging a guiding reference model.<n>Extensive experiments on AlpacaEval 2.0 and Arena-Hard v0.1 benchmarks demonstrate that Pre-DPO consistently improves the performance of both DPO and SimPO.
  arXiv  Detail & Related papers  (2025-04-22T12:39:30Z)
• Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimization [75.1240295759264]
  We propose an effective framework for Bridging and Modeling Correlations in pairwise data, named BMC.<n>We increase the consistency and informativeness of the pairwise preference signals through targeted modifications.<n>We identify that DPO alone is insufficient to model these correlations and capture nuanced variations.
  arXiv  Detail & Related papers  (2024-08-14T11:29:47Z)
• Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer [52.09480867526656]
  We identify the source of misalignment as a form of distributional shift and uncertainty in learning human preferences.<n>To mitigate overoptimization, we first propose a theoretical algorithm that chooses the best policy for an adversarially chosen reward model.<n>Using the equivalence between reward models and the corresponding optimal policy, the algorithm features a simple objective that combines a preference optimization loss and a supervised learning loss.
  arXiv  Detail & Related papers  (2024-05-26T05:38:50Z)
• Multi-Reference Preference Optimization for Large Language Models [56.84730239046117]
  We introduce a novel closed-form formulation for direct preference optimization using multiple reference models. The resulting algorithm, Multi-Reference Preference Optimization (MRPO), leverages broader prior knowledge from diverse reference models. Our experiments demonstrate that LLMs finetuned with MRPO generalize better in various preference data, regardless of data scarcity or abundance.
  arXiv  Detail & Related papers  (2024-05-26T00:29:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.