Inducing Robustness in a 2 Dimensional Direct Preference Optimization Paradigm

• URL: http://arxiv.org/abs/2505.01706v1
• Date: Sat, 03 May 2025 05:59:13 GMT
• Title: Inducing Robustness in a 2 Dimensional Direct Preference Optimization Paradigm
• Authors: Sarvesh Shashidhar, Ritik, Nachiketa Patil, Suraj Racha, Ganesh Ramakrishnan,
• Abstract summary: Direct Preference optimisation (DPO) has emerged as a powerful method for aligning Large Language Models with human preferences.<n>We investigate the performance of DPO using open-source preference datasets.<n>We propose an approach of incorporating segment-level score noise robustness to the 2D-DPO algorithm.
• Score: 16.66633426354087
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Direct Preference Optimisation (DPO) has emerged as a powerful method for aligning Large Language Models (LLMs) with human preferences, offering a stable and efficient alternative to approaches that use Reinforcement learning via Human Feedback. In this work, we investigate the performance of DPO using open-source preference datasets. One of the major drawbacks of DPO is that it doesn't induce granular scoring and treats all the segments of the responses with equal propensity. However, this is not practically true for human preferences since even "good" responses have segments that may not be preferred by the annotator. To resolve this, a 2-dimensional scoring for DPO alignment called 2D-DPO was proposed. We explore the 2D-DPO alignment paradigm and the advantages it provides over the standard DPO by comparing their win rates. It is observed that these methods, even though effective, are not robust to label/score noise. To counter this, we propose an approach of incorporating segment-level score noise robustness to the 2D-DPO algorithm. Along with theoretical backing, we also provide empirical verification in favour of the algorithm and introduce other noise models that can be present.

Related papers

• Token-Importance Guided Direct Preference Optimization [2.230951739798399]
  We propose a Token-Importance Guided Direct Preference Optimization (TI-DPO) to ensure that large language models generate outputs aligned with human preferences.<n> Experimental results show that TI-DPO achieves higher accuracy and stronger generative diversity, providing more stable and computationally efficient solutions.
  arXiv  Detail & Related papers  (2025-05-26T08:11:24Z)
• Active Learning for Direct Preference Optimization [59.84525302418018]
  Direct preference optimization (DPO) is a form of reinforcement learning from human feedback.<n>We propose an active learning framework for DPO, which can be applied to collect human feedback online or to choose the most informative subset of already collected feedback offline.
  arXiv  Detail & Related papers  (2025-03-03T00:36:31Z)
• C2-DPO: Constrained Controlled Direct Preference Optimization [22.730518243326394]
  Direct preference optimization (textttDPO) has emerged as a promising approach for solving the alignment problem in AI.<n>We show that textttDPO loss could be derived by starting from an alternative optimization problem that only defines the KL guardrail on in-sample responses.
  arXiv  Detail & Related papers  (2025-02-22T00:38:44Z)
• VPO: Leveraging the Number of Votes in Preference Optimization [5.200545764106177]
  We introduce a technique that leverages user voting data to better align with diverse subjective preferences. We develop the Vote-based Preference Optimization framework, which incorporates the number of votes on both sides to distinguish between controversial and obvious generation pairs.
  arXiv  Detail & Related papers  (2024-10-30T10:39:34Z)
• Uncertainty-Penalized Direct Preference Optimization [52.387088396044206]
  We develop a pessimistic framework for DPO by introducing preference uncertainty penalization schemes. The penalization serves as a correction to the loss which attenuates the loss gradient for uncertain samples. We show improved overall performance compared to vanilla DPO, as well as better completions on prompts from high-uncertainty chosen/rejected responses.
  arXiv  Detail & Related papers  (2024-10-26T14:24:37Z)
• 2D-DPO: Scaling Direct Preference Optimization with 2-Dimensional Supervision [28.742104593747033]
  We propose to extend the preference of DPO to two dimensions: segments and aspects. We develop a 2D-DPO framework, decomposing the overall objective into multi-segment and multi-aspect objectives.
  arXiv  Detail & Related papers  (2024-10-25T17:47:35Z)
• Accelerated Preference Optimization for Large Language Model Alignment [60.22606527763201]
  Reinforcement Learning from Human Feedback (RLHF) has emerged as a pivotal tool for aligning large language models (LLMs) with human preferences. Direct Preference Optimization (DPO) formulates RLHF as a policy optimization problem without explicitly estimating the reward function. We propose a general Accelerated Preference Optimization (APO) framework, which unifies many existing preference optimization algorithms.
  arXiv  Detail & Related papers  (2024-10-08T18:51:01Z)
• Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimization [75.1240295759264]
  We propose an effective framework for Bridging and Modeling Correlations in pairwise data, named BMC.<n>We increase the consistency and informativeness of the pairwise preference signals through targeted modifications.<n>We identify that DPO alone is insufficient to model these correlations and capture nuanced variations.
  arXiv  Detail & Related papers  (2024-08-14T11:29:47Z)
• Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization [45.6430987775264]
  This study addresses the challenge of noise in training datasets for Direct Preference Optimization (DPO)<n>We categorize noise into pointwise noise, which includes low-quality data points, and pairwise noise, which encompasses erroneous data pair associations.<n>We introduce Distributionally Robustifying DPO, which integrates pairwise robustness by optimizing against worst-case pairwise scenarios.
  arXiv  Detail & Related papers  (2024-07-10T17:48:25Z)
• D2PO: Discriminator-Guided DPO with Response Evaluation Models [63.71853401569461]
  We propose D2PO, discriminator-guided DPO, for the online setting where preferences are being collected throughout learning. As we collect gold preferences, we use these not only to train our policy, but to train a discriminative response evaluation model to silver-label even more synthetic data for policy training. We show conditions under which silver labeling is most helpful: it is most effective when training the policy with DPO, outperforming traditional PPO, and benefits from maintaining a separate discriminator from the policy model.
  arXiv  Detail & Related papers  (2024-05-02T17:44:41Z)
• ROPO: Robust Preference Optimization for Large Language Models [59.10763211091664]
  We propose an iterative alignment approach that integrates noise-tolerance and filtering of noisy samples without the aid of external models. Experiments on three widely-used datasets with Mistral-7B and Llama-2-7B demonstrate that ROPO significantly outperforms existing preference alignment methods.
  arXiv  Detail & Related papers  (2024-04-05T13:58:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.