Disentangling Optimization Scale from Preference Scale in DPO

• URL: http://arxiv.org/abs/2608.27032v1
• Date: Thu, 27 Aug 2026 12:21:14 GMT
• Title: Disentangling Optimization Scale from Preference Scale in DPO
• Abstract summary: We show that $$ entangles two distinct roles: it governs the effective inverse preference-noise scale and simultaneously rescales the optimization dynamics.<n>We propose a centered-softplus reformulation that is argmin-equivalent to DPO for $>0$, while making the inverse preference-noise-scale and learning-rate effects explicit and independently tunable.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Direct Preference Optimization (DPO) is a widely used objective for aligning language models from preference data, with the coefficient $β$ commonly interpreted as controlling the KL constraint to a reference policy. We show that $β$ entangles two distinct roles: it governs the effective inverse preference-noise scale and simultaneously rescales the optimization dynamics, coupling this scale with the effective step size. As a consequence, at a fixed learning rate the achieved policy deviation is non-monotone in $β$: it vanishes in a dead zone at small $β$, reaches a peak at an intermediate value, and decreases again for larger $β$. Moreover, standard DPO loss values are not comparable across $β$: runs with nearly identical loss curves can differ several-fold in KL divergence from the reference model. This entanglement obscures the role of $β$, increases sensitivity to hyperparameter choices, and complicates learning-rate scheduling. We propose a centered-softplus reformulation that is argmin-equivalent to DPO for $β>0$, while making the inverse preference-noise-scale and learning-rate effects explicit and independently tunable. The normalized centered-softplus objective also admits a continuous $β\to0$ endpoint that reduces to a linear preference-margin objective.

Related papers

• $β$-OPSD: Deriving with Policy Optimization, Training with Self-Distillation [85.07190808882523]
  On-policy self-distillation is a promising approach to improve reasoning language models.<n>We show that $$-OPSD consistently outperforms vanilla OPSD.
  arXiv  Detail & Related papers  (2026-07-30T17:41:16Z)
• How Neural Reward Models Learn Features for Policy Optimization: A Single-Index Analysis [53.063298916923976]
  We study the feedback in a Gaussian single-index model with $r*(x) = *(langle *, xrangle)$ and $x sim N(0, I_d)$.<n>We analyze a two-stage neural reward model that first learns the hidden direction $*$ from reward-weighted samples and then fits the readout layer by weighted ridge regression.
  arXiv  Detail & Related papers  (2026-05-23T22:00:38Z)
• $ξ$-DPO: Direct Preference Optimization via Ratio Reward Margin [16.49252561546099]
  We argue that the margin formulation in SimPO is not easily interpretable across datasets with different reward gap structures.<n>We propose $$-DPO: Direct preference optimization via ratio reward margin.
  arXiv  Detail & Related papers  (2026-05-09T03:48:16Z)
• Mitigating Mismatch within Reference-based Preference Optimization [55.07698254211876]
  Direct Preference Optimization (DPO) has become the de facto standard for offline preference alignment of large language models.<n>DPO weighs each update relative to a reference, which stabilizes the training by regularizing the updates within a trusted region.<n>This reliance becomes problematic for pessimistic pairs, where the reference model prefers the rejected response.<n>We modify DPO to treat the reference as neutral when it is pessimistic by replacing $_-_mathrmref$ with $_-max0,_mathrmref$.
  arXiv  Detail & Related papers  (2026-02-12T12:55:51Z)
• The Viscosity of Logic: Phase Transitions and Hysteresis in DPO Alignment [0.0]
  Direct Preference Optimization (DPO) is often tuned as if increasing alignment pressure (controlled by $$) yields progressively "better" behavior.<n>We treat $$ as a control parameter and densely sweep it for three 7B open-weight families under a fixed DPO recipe.
  arXiv  Detail & Related papers  (2026-01-24T02:19:20Z)
• DRO-REBEL: Distributionally Robust Relative-Reward Regression for Fast and Efficient LLM Alignment [0.0]
  Reinforcement learning with human feedback (RLHF) has become crucial for aligning Large Language Models (LLMs) with human intent.<n>We introduce DRO-REBEL, a unified family of robust REBEL updates with type-$p$ Wasserstein, KL, and $chi2$ ambiguity sets.<n>Using Fenchel duality, each update reduces to a simple relative-reward regression, preserving scalability and avoiding PPO-style clipping or auxiliary value networks.
  arXiv  Detail & Related papers  (2025-09-23T14:49:48Z)
• Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences? [20.004349891563706]
  After pre-training, large language models are aligned with human preferences based on pairwise comparisons.<n>We introduce an alignment method's distortion: the worst-case ratio between the optimal achievable average utility, and the average utility of the learned policy.
  arXiv  Detail & Related papers  (2025-05-29T17:59:20Z)
• FedSVD: Adaptive Orthogonalization for Private Federated Learning with LoRA [68.44043212834204]
  Low-Rank Adaptation (LoRA) is widely used for efficient fine-tuning of language models in learning (FL)<n>Low-Rank Adaptation (LoRA) is widely used for efficient fine-tuning of language models in learning (FL)
  arXiv  Detail & Related papers  (2025-05-19T07:32:56Z)
• RePO: Understanding Preference Learning Through ReLU-Based Optimization [66.098833436503]
  We propose ReLU-based Preference Optimization (RePO), a streamlined algorithm that eliminates $beta$ via two advances.<n>RePO is characterized as SimPO's limiting case ($beta to infty$), where the logistic weighting collapses to binary thresholding.<n> Empirical results on AlpacaEval 2 and Arena-Hard show that RePO outperforms DPO and SimPO across multiple base models.
  arXiv  Detail & Related papers  (2025-03-10T15:11:07Z)
• KL Penalty Control via Perturbation for Direct Preference Optimization [55.24971559149686]
  We propose $varepsilon$-Direct Preference Optimization ($varepsilon$-DPO), which allows adaptive control of the KL penalty strength $beta$ for each preference pair.<n> Experimental results show that the simple criterion of $varepsilon$-DPO for KL penalty relaxation significantly improves DPO compared to most existing direct alignment algorithms.
  arXiv  Detail & Related papers  (2025-02-18T06:44:10Z)
• AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization [45.46582930202524]
  $alpha$-DPO is an adaptive preference optimization algorithm for large language models.<n>It balances the policy model and the reference model to achieve personalized reward margins.<n>It consistently outperforms DPO and SimPO across various model settings.
  arXiv  Detail & Related papers  (2024-10-14T04:29:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.