AdaDPO: Self-Adaptive Direct Preference Optimization with Balanced Gradient Updates

• URL: http://arxiv.org/abs/2605.28440v1
• Date: Wed, 27 May 2026 13:05:49 GMT
• Title: AdaDPO: Self-Adaptive Direct Preference Optimization with Balanced Gradient Updates
• Abstract summary: We propose a Self-Adaptive variant of the DPO algorithm that introduces per-preference-pair, stop-gradient-based coefficients.<n>AdaDPO is constructed to enforce equality of gradient magnitudes between preferred and dispreferred probabilities.<n>Because it operates purely at the loss level, AdaDPO can be dropped into existing preference-based alignment pipelines.
• Score: 0.03999851878220877
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: DPO has become a widely adopted alternative to RLHF for aligning LLMs with human preferences, eliminating the need for a separate reward model or RL loop. Recent theoretical analysis uncovers an asymmetric gradient behavior in DPO: the loss suppresses dispreferred responses substantially faster than it promotes preferred ones, causing the model to learn to avoid bad answers rather than to generate good ones. We propose AdaDPO, a Self-Adaptive variant of the DPO algorithm that introduces per-preference-pair, stop-gradient-based coefficients derived directly from the policy model's generation probabilities, with the reference model's probabilities as an optional component. AdaDPO is constructed to enforce equality of gradient magnitudes between preferred and dispreferred probabilities; the practical implementation balances per-token gradients and applies a numerical clipping bound for stability, while retaining DPO's original hyperparameter structure. On Llama-3-8B-Instruct trained on UltraFeedback under a SimPO similar setup, AdaDPO consistently outperforms DPO on AlpacaEval 2: it achieves higher length-controlled win rates (LC) in 81% of hyperparameter combinations, attains the global best LC (48.3%) and raw win rate (46.1%), and enlarges the LC-over-WR margin in 88% of combinations, indicating effective mitigation of length bias. Additional analyses on KL divergence, reward margin, and reward accuracy confirm that AdaDPO rectifies the gradient imbalance and yields more efficient optimization. Because it operates purely at the loss level, AdaDPO can be dropped into existing preference-based alignment pipelines without changing data collection or model architectures. The method requires only a few lines of code, and the same self-adaptive principle generalizes to a broad family of pairwise contrastive preference losses including SimPO, R-DPO, IPO, CPO, and ORPO.

Related papers

• Conditional Equivalence of DPO and RLHF: Implicit Assumption, Failure Modes, and Provable Alignment [51.18269946911088]
  Direct Preference Optimization (DPO) has emerged as a popular alternative to Reinforcement Learning from Human Feedback (RLHF)<n>We show that DPO and RLHF optimize fundamentally different objectives in such cases.<n>We introduce Constrained Preference Optimization (CPO), augmenting RLHF with constraints for provable alignment.<n>Our theoretical analysis establishes when DPOs' guarantees hold and provides solutions preserving simplicity with provable alignment.
  arXiv  Detail & Related papers  (2026-05-20T07:26:22Z)
• wDPO: Winsorized Direct Preference Optimization for Robust LLM Alignment [48.487557157323664]
  In practice, preference data are often noisy.<n>Existing robust variants of DPO mainly rely on uniform objective modifications or global reweighting.<n>We show that robust preference alignment benefits from addressing different noise types with targeted interventions.
  arXiv  Detail & Related papers  (2026-03-07T13:30:53Z)
• Unbiased Dynamic Pruning for Efficient Group-Based Policy Optimization [60.87651283510059]
  Group Relative Policy Optimization (GRPO) effectively scales LLM reasoning but incurs prohibitive computational costs.<n>We propose Dynamic Pruning Policy Optimization (DPPO), a framework that enables dynamic pruning while preserving unbiased gradient estimation.<n>To mitigate the data sparsity induced by pruning, we introduce Dense Prompt Packing, a window-based greedy strategy.
  arXiv  Detail & Related papers  (2026-03-04T14:48:53Z)
• Rethinking the Trust Region in LLM Reinforcement Learning [72.25890308541334]
  Proximal Policy Optimization (PPO) serves as the de facto standard algorithm for Large Language Models (LLMs)<n>We propose Divergence Proximal Policy Optimization (DPPO), which substitutes clipping with a more principled constraint.<n>DPPO achieves superior training and efficiency compared to existing methods, offering a more robust foundation for RL-based fine-tuning.
  arXiv  Detail & Related papers  (2026-02-04T18:59:04Z)
• BPO: Revisiting Preference Modeling in Direct Preference Optimization [13.243174453617064]
  Direct Preference Optimization (DPO) has emerged as a popular method for aligning Large Language Models (LLMs) with human preferences.<n>DPO effectively preserves the relative ordering between chosen and rejected responses through pairwise ranking losses.<n>It often neglects absolute reward magnitudes, leading to poor performance.<n>We propose Balanced Preference Optimization (BPO), a novel framework that balances the optimization of chosen and rejected responses.
  arXiv  Detail & Related papers  (2025-06-04T04:21:01Z)
• Uncertainty-Penalized Direct Preference Optimization [52.387088396044206]
  We develop a pessimistic framework for DPO by introducing preference uncertainty penalization schemes. The penalization serves as a correction to the loss which attenuates the loss gradient for uncertain samples. We show improved overall performance compared to vanilla DPO, as well as better completions on prompts from high-uncertainty chosen/rejected responses.
  arXiv  Detail & Related papers  (2024-10-26T14:24:37Z)
• AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization [45.46582930202524]
  $alpha$-DPO is an adaptive preference optimization algorithm for large language models.<n>It balances the policy model and the reference model to achieve personalized reward margins.<n>It consistently outperforms DPO and SimPO across various model settings.
  arXiv  Detail & Related papers  (2024-10-14T04:29:57Z)
• Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer [52.09480867526656]
  We identify the source of misalignment as a form of distributional shift and uncertainty in learning human preferences.<n>To mitigate overoptimization, we first propose a theoretical algorithm that chooses the best policy for an adversarially chosen reward model.<n>Using the equivalence between reward models and the corresponding optimal policy, the algorithm features a simple objective that combines a preference optimization loss and a supervised learning loss.
  arXiv  Detail & Related papers  (2024-05-26T05:38:50Z)
• RS-DPO: A Hybrid Rejection Sampling and Direct Preference Optimization Method for Alignment of Large Language Models [7.676477609461592]
  Reinforcement learning from human feedback (RLHF) has been extensively employed to align large language models with user intent. DPO relies on contrastive responses generated from human annotator and alternative LLM, instead of the policy model. In this paper, we address both challenges by systematically combining sampling rejection (RS) and DPO. Our proposed method effectively fine-tunes LLMs with limited resource environments, leading to improved alignment with user intent.
  arXiv  Detail & Related papers  (2024-02-15T16:00:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.