Gradient Imbalance in Direct Preference Optimization

• URL: http://arxiv.org/abs/2502.20847v1
• Date: Fri, 28 Feb 2025 08:47:03 GMT
• Title: Gradient Imbalance in Direct Preference Optimization
• Authors: Qinwei Ma, Jingzhe Shi, Can Jin, Jenq-Neng Hwang, Serge Belongie, Lei Li,
• Abstract summary: We propose Balanced-DPO, a simple yet effective modification to the DPO objective that introduces a computationally efficient gradient reweighting mechanism.<n>Our experiments demonstrate the effectiveness of Balanced-DPO, validating the theoretical findings and confirming that addressing gradient imbalance is key to improving DPO's performance.
• Score: 26.964127989679596
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Direct Preference Optimization (DPO) has been proposed as a promising alternative to Proximal Policy Optimization (PPO) based Reinforcement Learning with Human Feedback (RLHF). However, empirical evaluations consistently reveal suboptimal performance in DPO compared to common RLHF pipelines. In this work, we conduct a systematic analysis of DPO's training dynamics and identify gradient imbalance as a critical limitation. We demonstrate theoretically and empirically that this imbalance perturbs optimization trajectories, destabilizes learning, and induces suboptimal convergence. To address this issue, we propose Balanced-DPO, a simple yet effective modification to the DPO objective that introduces a computationally efficient gradient reweighting mechanism. Our experiments demonstrate the effectiveness of Balanced-DPO, validating the theoretical findings and confirming that addressing gradient imbalance is key to improving DPO's performance, highlighting a promising direction for future research.

Related papers

• Rethinking DPO: The Role of Rejected Responses in Preference Misalignment [4.760537994346813]
  We propose Bounded-DPO (BDPO), a novel method that bounds the influence of rejected responses while maintaining the original optimization structure of DPO.<n>We demonstrate that BDPO achieves a balanced optimization of the chosen and rejected responses, outperforming existing algorithms.
  arXiv  Detail & Related papers  (2025-06-15T05:32:07Z)
• Understanding the Impact of Sampling Quality in Direct Preference Optimization [2.1624199216631625]
  We first analyze how distribution of responses influences policy updates during gradient descent.<n>We then design a simplified yet well-structured alignment model as a proxy, and develop quantitative results showing how more frequent high-quality responses amplify the gradient signal.
  arXiv  Detail & Related papers  (2025-06-03T18:12:40Z)
• Understanding the Performance Gap in Preference Learning: A Dichotomy of RLHF and DPO [51.22869332661607]
  We decompose the performance gap between reinforcement learning from human feedback and direct preference optimization under a representation gap.<n>We show that RLHF, DPO, or online DPO can outperform one another depending on the type of model mis-specifications.
  arXiv  Detail & Related papers  (2025-05-26T09:54:02Z)
• On the Effect of Negative Gradient in Group Relative Deep Reinforcement Optimization [52.76330545825083]
  Reinforcement learning (RL) has become popular in enhancing the reasoning capabilities of large language models (LLMs)<n>We identify a previously unrecognized phenomenon we term Lazy Likelihood Displacement (LLD), wherein the likelihood of correct responses marginally increases or even decreases during training.<n>We develop a method called NTHR, which downweights penalties on tokens contributing to the LLD. Unlike prior DPO-based approaches, NTHR takes advantage of GRPO's group-based structure, using correct responses as anchors to identify influential tokens.
  arXiv  Detail & Related papers  (2025-05-24T18:58:51Z)
• SGDPO: Self-Guided Direct Preference Optimization for Language Model Alignment [46.55132297735257]
  We propose a novel Self-Guided Direct Preference Optimization algorithm, i.e., SGDPO, which incorporates a pilot term to steer the gradient flow during the optimization process.<n>We provide a detailed theoretical analysis of our proposed method and elucidate its operational mechanism.
  arXiv  Detail & Related papers  (2025-05-18T14:19:23Z)
• A Survey of Direct Preference Optimization [103.59317151002693]
  Large Language Models (LLMs) have demonstrated unprecedented generative capabilities. Their alignment with human values remains critical for ensuring helpful and harmless deployments. Direct Preference Optimization (DPO) has recently gained prominence as a streamlined alternative.
  arXiv  Detail & Related papers  (2025-03-12T08:45:15Z)
• A Simple and Effective Reinforcement Learning Method for Text-to-Image Diffusion Fine-tuning [61.403275660120606]
  Reinforcement learning (RL)-based fine-tuning has emerged as a powerful approach for aligning diffusion models with black-box objectives. We propose leave-one-out PPO (LOOP), a novel RL for diffusion fine-tuning method. Our results demonstrate that LOOP effectively improves diffusion models on various black-box objectives, and achieves a better balance between computational efficiency and performance.
  arXiv  Detail & Related papers  (2025-03-02T13:43:53Z)
• Entropy Controllable Direct Preference Optimization [3.536605202672355]
  We propose a simple modification to DPO, H-DPO, which allows for control over the entropy of the resulting policy. In our experiments, we show that H-DPO outperformed DPO across various tasks, demonstrating superior results in pass@$k$ evaluations for mathematical tasks.
  arXiv  Detail & Related papers  (2024-11-12T07:09:44Z)
• Uncertainty-Penalized Direct Preference Optimization [52.387088396044206]
  We develop a pessimistic framework for DPO by introducing preference uncertainty penalization schemes. The penalization serves as a correction to the loss which attenuates the loss gradient for uncertain samples. We show improved overall performance compared to vanilla DPO, as well as better completions on prompts from high-uncertainty chosen/rejected responses.
  arXiv  Detail & Related papers  (2024-10-26T14:24:37Z)
• $α$-DPO: Adaptive Reward Margin is What Direct Preference Optimization Needs [45.46582930202524]
  $alpha$-DPO is an adaptive preference optimization algorithm for large language models. It balances the policy model and the reference model to achieve personalized reward margins. It consistently outperforms DPO and SimPO across various model settings.
  arXiv  Detail & Related papers  (2024-10-14T04:29:57Z)
• Accelerated Preference Optimization for Large Language Model Alignment [60.22606527763201]
  Reinforcement Learning from Human Feedback (RLHF) has emerged as a pivotal tool for aligning large language models (LLMs) with human preferences. Direct Preference Optimization (DPO) formulates RLHF as a policy optimization problem without explicitly estimating the reward function. We propose a general Accelerated Preference Optimization (APO) framework, which unifies many existing preference optimization algorithms.
  arXiv  Detail & Related papers  (2024-10-08T18:51:01Z)
• 3D-Properties: Identifying Challenges in DPO and Charting a Path Forward [17.27880657597116]
  We revisit DPO, analyzing its theoretical foundations and empirical performance.<n>We identify three key properties, termed 3D properties, that emerge from DPO's learning process.<n>We propose simple regularization techniques that improve training stability and performance.
  arXiv  Detail & Related papers  (2024-06-11T14:59:24Z)
• Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer [52.09480867526656]
  We identify the source of misalignment as a form of distributional shift and uncertainty in learning human preferences.<n>To mitigate overoptimization, we first propose a theoretical algorithm that chooses the best policy for an adversarially chosen reward model.<n>Using the equivalence between reward models and the corresponding optimal policy, the algorithm features a simple objective that combines a preference optimization loss and a supervised learning loss.
  arXiv  Detail & Related papers  (2024-05-26T05:38:50Z)
• Towards Analyzing and Understanding the Limitations of DPO: A Theoretical Perspective [25.34250859820326]
  We provide an analytical framework using the field theory to analyze the optimization process of DPO. We find that the DPO loss function decreases the probability of producing human dispreferred data at a faster rate than it increases the probability of producing preferred data.
  arXiv  Detail & Related papers  (2024-04-06T13:24:37Z)
• Overcoming Reward Overoptimization via Adversarial Policy Optimization with Lightweight Uncertainty Estimation [46.61909578101735]
  Adversarial Policy Optimization (AdvPO) is a novel solution to the pervasive issue of reward over-optimization in Reinforcement Learning from Human Feedback. In this paper, we introduce a lightweight way to quantify uncertainties in rewards, relying solely on the last layer embeddings of the reward model.
  arXiv  Detail & Related papers  (2024-03-08T09:20:12Z)
• Statistical Rejection Sampling Improves Preference Optimization [42.57245965632205]
  We introduce a novel approach to source preference data from the target optimal policy using rejection sampling. We also propose a unified framework that enhances the loss functions used in both Sequence Likelihood (SLiC) and Direct Preference Optimization (DPO) from a preference modeling standpoint.
  arXiv  Detail & Related papers  (2023-09-13T01:07:25Z)
• Fine-Tuning Language Models with Advantage-Induced Policy Alignment [80.96507425217472]
  We propose a novel algorithm for aligning large language models to human preferences. We show that it consistently outperforms PPO in language tasks by a large margin. We also provide a theoretical justification supporting the design of our loss function.
  arXiv  Detail & Related papers  (2023-06-04T01:59:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.