DPO-Shift: Shifting the Distribution of Direct Preference Optimization

• URL: http://arxiv.org/abs/2502.07599v1
• Date: Tue, 11 Feb 2025 14:49:44 GMT
• Title: DPO-Shift: Shifting the Distribution of Direct Preference Optimization
• Authors: Xiliang Yang, Feng Jiang, Qianen Zhang, Lei Zhao, Xiao Li,
• Abstract summary: We introduce method to controllably shift the distribution of the chosen probability. We show that method exhibits a fundamental trade-off between improving the chosen probability and sacrificing the reward margin.
• Score: 13.261638548369188
• License:
• Abstract: Direct Preference Optimization (DPO) and its variants have become increasingly popular for aligning language models with human preferences. These methods aim to teach models to better distinguish between chosen (or preferred) and rejected (or dispreferred) responses. However, prior research has identified that the probability of chosen responses often decreases during training, and this phenomenon is known as likelihood displacement. To tackle this challenge, in this work we introduce \method to controllably shift the distribution of the chosen probability. Then, we show that \method exhibits a fundamental trade-off between improving the chosen probability and sacrificing the reward margin, as supported by both theoretical analysis and experimental validation. Furthermore, we demonstrate the superiority of \method over DPO on downstream tasks such as MT-Bench and a designed win rate experiment. We believe this study shows that the likelihood displacement issue of DPO can be effectively mitigated with a simple, theoretically grounded solution. Our code is available at https://github.com/Meaquadddd/DPO-Shift.

Related papers

• Distributionally Robust Coreset Selection under Covariate Shift [14.979902937312099]
  Coreset selection is an approach to reducing training data. It is challenging to select an effective subset of training data that performs well across all deployment scenarios. We propose Distributionally Robust Coreset Selection (DRCS) for this purpose.
  arXiv  Detail & Related papers  (2025-01-24T05:36:53Z)
• Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization [60.176008034221404]
  Direct Preference Optimization (DPO) and its variants are increasingly used for aligning language models with human preferences. Prior work has observed that the likelihood of preferred responses often decreases during training. We demonstrate that likelihood displacement can be catastrophic, shifting probability mass from preferred responses to responses with an opposite meaning.
  arXiv  Detail & Related papers  (2024-10-11T14:22:44Z)
• 3D-Properties: Identifying Challenges in DPO and Charting a Path Forward [17.27880657597116]
  We revisit DPO, analyzing its theoretical foundations and empirical performance. We identify three key properties, termed 3D properties, that emerge from DPO's learning process. We propose simple regularization techniques that improve training stability and performance.
  arXiv  Detail & Related papers  (2024-06-11T14:59:24Z)
• Robust Preference Optimization through Reward Model Distillation [68.65844394615702]
  Language model (LM) post-training involves maximizing a reward function that is derived from preference annotations. DPO is a popular offline alignment method that trains a policy directly on preference data without the need to train a reward model or apply reinforcement learning. We analyze this phenomenon and propose distillation to get a better proxy for the true preference distribution over generation pairs.
  arXiv  Detail & Related papers  (2024-05-29T17:39:48Z)
• Rejection via Learning Density Ratios [50.91522897152437]
  Classification with rejection emerges as a learning paradigm which allows models to abstain from making predictions. We propose a different distributional perspective, where we seek to find an idealized data distribution which maximizes a pretrained model's performance. Our framework is tested empirically over clean and noisy datasets.
  arXiv  Detail & Related papers  (2024-05-29T01:32:17Z)
• Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer [52.09480867526656]
  We identify the source of misalignment as a form of distributional shift and uncertainty in learning human preferences. To mitigate overoptimization, we first propose a theoretical algorithm that chooses the best policy for an adversarially chosen reward model. Using the equivalence between reward models and the corresponding optimal policy, the algorithm features a simple objective that combines a preference optimization loss and a supervised learning loss.
  arXiv  Detail & Related papers  (2024-05-26T05:38:50Z)
• Source-Free Unsupervised Domain Adaptation with Hypothesis Consolidation of Prediction Rationale [53.152460508207184]
  Source-Free Unsupervised Domain Adaptation (SFUDA) is a challenging task where a model needs to be adapted to a new domain without access to target domain labels or source domain data. This paper proposes a novel approach that considers multiple prediction hypotheses for each sample and investigates the rationale behind each hypothesis. To achieve the optimal performance, we propose a three-step adaptation process: model pre-adaptation, hypothesis consolidation, and semi-supervised learning.
  arXiv  Detail & Related papers  (2024-02-02T05:53:22Z)
• Distributionally Robust Post-hoc Classifiers under Prior Shifts [31.237674771958165]
  We investigate the problem of training models that are robust to shifts caused by changes in the distribution of class-priors or group-priors. We present an extremely lightweight post-hoc approach that performs scaling adjustments to predictions from a pre-trained model.
  arXiv  Detail & Related papers  (2023-09-16T00:54:57Z)
• KL Guided Domain Adaptation [88.19298405363452]
  Domain adaptation is an important problem and often needed for real-world applications. A common approach in the domain adaptation literature is to learn a representation of the input that has the same distributions over the source and the target domain. We show that with a probabilistic representation network, the KL term can be estimated efficiently via minibatch samples.
  arXiv  Detail & Related papers  (2021-06-14T22:24:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.