Rethinking DPO: The Role of Rejected Responses in Preference Misalignment

• URL: http://arxiv.org/abs/2506.12725v1
• Date: Sun, 15 Jun 2025 05:32:07 GMT
• Title: Rethinking DPO: The Role of Rejected Responses in Preference Misalignment
• Abstract summary: We propose Bounded-DPO (BDPO), a novel method that bounds the influence of rejected responses while maintaining the original optimization structure of DPO.<n>We demonstrate that BDPO achieves a balanced optimization of the chosen and rejected responses, outperforming existing algorithms.
• Score: 4.760537994346813
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Direct Preference Optimization (DPO) is a simple and efficient framework that has attracted substantial attention. However, it often struggles to meet its primary objectives -- increasing the generation probability of chosen responses while reducing that of rejected responses -- due to the dominant influence of rejected responses on the loss function. This imbalance leads to suboptimal performance in promoting preferred responses. In this work, we systematically analyze the limitations of DPO and existing algorithms designed to achieve the objectives stated above. To address these limitations, we propose Bounded-DPO (BDPO), a novel method that bounds the influence of rejected responses while maintaining the original optimization structure of DPO. Through theoretical analysis and empirical evaluations, we demonstrate that BDPO achieves a balanced optimization of the chosen and rejected responses, outperforming existing algorithms.

Related papers

• RLVR without Ineffective Samples: Group Prioritized Off-Policy Optimization for LLM Reasoning [49.04912820721943]
  Group Prioritized Off-Policy Optimization (POPO) is a framework that exploits effective training batches without additional rollout overhead.<n>POPO comprises two key components: prioritized group replay and decoupled off-policy optimization.<n>POPO substantially accelerates RL finetuning and achieves strong reasoning performance with significantly fewer rollouts.
  arXiv  Detail & Related papers  (2026-05-31T15:06:38Z)
• Conditional Equivalence of DPO and RLHF: Implicit Assumption, Failure Modes, and Provable Alignment [51.18269946911088]
  Direct Preference Optimization (DPO) has emerged as a popular alternative to Reinforcement Learning from Human Feedback (RLHF)<n>We show that DPO and RLHF optimize fundamentally different objectives in such cases.<n>We introduce Constrained Preference Optimization (CPO), augmenting RLHF with constraints for provable alignment.<n>Our theoretical analysis establishes when DPOs' guarantees hold and provides solutions preserving simplicity with provable alignment.
  arXiv  Detail & Related papers  (2026-05-20T07:26:22Z)
• Bounded Ratio Reinforcement Learning [62.09379476369526]
  This paper introduces the Bounded Ratio Reinforcement Learning (BRRL) framework to solve on-policy reinforcement learning problems.<n>We develop a policy optimization algorithm called Bounded Policy Optimization (BPO) that minimizes an advantage-weighted divergence between the policy and the analytic optimal solution from BRRL.<n> Notably, our framework also provides a new theoretical lens to interpret the success of the PPO loss, and connects trust region policy optimization and the Cross-Entropy Method (CEM).
  arXiv  Detail & Related papers  (2026-04-20T17:59:01Z)
• Provable and Practical In-Context Policy Optimization for Self-Improvement [49.670847804409874]
  We study test-time scaling, where a model improves its answer through multi-round self-reflection at inference.<n>We introduce In-Context Policy Optimization (ICPO), in which an agent optimize its response in context using self-assessed or externally observed rewards without modifying its parameters.<n>We propose Minimum-Entropy ICPO (ME-ICPO), a practical algorithm that iteratively uses its response and self-assessed reward to refine its response in-context at inference time.
  arXiv  Detail & Related papers  (2026-03-02T00:21:50Z)
• What Is Preference Optimization Doing, How and Why? [73.78865043839939]
  A common belief is that DPO is supervised learning while PPO is reinforcement learning, yet deeper analyses for the reasons underlying these differences remain lacking.<n>First, we examine the target directions of gradient-based updates and find that DPO follows stable targets, whereas PPO follows dynamic targets that balance exploration and exploitation.<n>Second, we examine the roles of positive learning, negative learning, and loss reweighting, which are three key components in PO methods.
  arXiv  Detail & Related papers  (2025-11-30T08:27:59Z)
• What Matters in Data for DPO? [6.208229499655634]
  Direct Preference Optimization (DPO) has emerged as a simple and effective approach for aligning large language models with human preferences.<n>We provide a systematic study of how preference data distribution influences DPO, from both theoretical and empirical perspectives.<n>We show that the quality of chosen responses plays a dominant role in optimizing the DPO objective, while the quality of rejected responses may have relatively limited impact.
  arXiv  Detail & Related papers  (2025-08-23T16:00:30Z)
• Explicit Preference Optimization: No Need for an Implicit Reward Model [18.225409932618657]
  Direct preference optimization (DPO) and its offshoots circumvent the need for a separate reward training step.<n>We show that DPO-based objectives are nonetheless subject to sub-optimal regularization and counter-intuitive artifacts.
  arXiv  Detail & Related papers  (2025-06-09T07:11:01Z)
• On the Effect of Negative Gradient in Group Relative Deep Reinforcement Optimization [52.76330545825083]
  Reinforcement learning (RL) has become popular in enhancing the reasoning capabilities of large language models (LLMs)<n>We identify a previously unrecognized phenomenon we term Lazy Likelihood Displacement (LLD), wherein the likelihood of correct responses marginally increases or even decreases during training.<n>We develop a method called NTHR, which downweights penalties on tokens contributing to the LLD. Unlike prior DPO-based approaches, NTHR takes advantage of GRPO's group-based structure, using correct responses as anchors to identify influential tokens.
  arXiv  Detail & Related papers  (2025-05-24T18:58:51Z)
• DisCO: Reinforcing Large Reasoning Models with Discriminative Constrained Optimization [55.06360285372418]
  Group Relative Policy Optimization is a reinforcement learning method for large reasoning models (LRMs)<n>In this work, we analyze the GRPO objective under a binary reward setting and reveal an inherent limitation of question-level difficulty bias.<n>We introduce a new Discriminative Constrained Optimization framework for reinforcing LRMs, grounded in the principle of discriminative learning.
  arXiv  Detail & Related papers  (2025-05-18T11:08:32Z)
• A Simple and Effective Reinforcement Learning Method for Text-to-Image Diffusion Fine-tuning [61.403275660120606]
  Reinforcement learning (RL)-based fine-tuning has emerged as a powerful approach for aligning diffusion models with black-box objectives.<n>We propose leave-one-out PPO (LOOP), a novel RL for diffusion fine-tuning method.<n>Our results demonstrate that LOOP effectively improves diffusion models on various black-box objectives, and achieves a better balance between computational efficiency and performance.
  arXiv  Detail & Related papers  (2025-03-02T13:43:53Z)
• Gradient Imbalance in Direct Preference Optimization [26.964127989679596]
  We propose Balanced-DPO, a simple yet effective modification to the DPO objective that introduces a computationally efficient gradient reweighting mechanism.<n>Our experiments demonstrate the effectiveness of Balanced-DPO, validating the theoretical findings and confirming that addressing gradient imbalance is key to improving DPO's performance.
  arXiv  Detail & Related papers  (2025-02-28T08:47:03Z)
• Uncertainty-Penalized Direct Preference Optimization [52.387088396044206]
  We develop a pessimistic framework for DPO by introducing preference uncertainty penalization schemes. The penalization serves as a correction to the loss which attenuates the loss gradient for uncertain samples. We show improved overall performance compared to vanilla DPO, as well as better completions on prompts from high-uncertainty chosen/rejected responses.
  arXiv  Detail & Related papers  (2024-10-26T14:24:37Z)
• 3D-Properties: Identifying Challenges in DPO and Charting a Path Forward [17.27880657597116]
  We revisit DPO, analyzing its theoretical foundations and empirical performance.<n>We identify three key properties, termed 3D properties, that emerge from DPO's learning process.<n>We propose simple regularization techniques that improve training stability and performance.
  arXiv  Detail & Related papers  (2024-06-11T14:59:24Z)
• Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer [52.09480867526656]
  We identify the source of misalignment as a form of distributional shift and uncertainty in learning human preferences.<n>To mitigate overoptimization, we first propose a theoretical algorithm that chooses the best policy for an adversarially chosen reward model.<n>Using the equivalence between reward models and the corresponding optimal policy, the algorithm features a simple objective that combines a preference optimization loss and a supervised learning loss.
  arXiv  Detail & Related papers  (2024-05-26T05:38:50Z)
• Overcoming Reward Overoptimization via Adversarial Policy Optimization with Lightweight Uncertainty Estimation [46.61909578101735]
  Adversarial Policy Optimization (AdvPO) is a novel solution to the pervasive issue of reward over-optimization in Reinforcement Learning from Human Feedback. In this paper, we introduce a lightweight way to quantify uncertainties in rewards, relying solely on the last layer embeddings of the reward model.
  arXiv  Detail & Related papers  (2024-03-08T09:20:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.