Towards Disentangled Preference Optimization Dynamics Beyond Likelihood Displacement

• URL: http://arxiv.org/abs/2604.18239v2
• Date: Sat, 25 Apr 2026 16:04:01 GMT
• Title: Towards Disentangled Preference Optimization Dynamics Beyond Likelihood Displacement
• Abstract summary: We present a unified emphincentive-score decomposition of preference optimization.<n>We identify the emphdisentanglement band (DB), a simple, testable condition that characterizes when training can avoid likelihood displacement.<n>We propose a plug-and-play emphreward calibration (RC) that adaptively rebalances chosen versus rejected updates to satisfy the DB and likelihood displacement.
• Score: 33.80669933764735
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Preference optimization is widely used to align large language models (LLMs) with human preferences. However, many margin-based objectives suppress the chosen response along with the rejected one, a phenomenon known as likelihood displacement, and no general mechanism currently prevents this across objectives. We bridge this gap by presenting a unified \emph{incentive-score decomposition} of preference optimization, revealing that diverse objectives share identical local update directions and differ only in their scalar weighting coefficients. Building on this decomposition, by analyzing the dynamics of the chosen/rejected likelihoods, we identify the \emph{disentanglement band} (DB), a simple, testable condition that characterizes when training can avoid likelihood displacement by realizing the preferred pathway: suppressing the loser while maintaining the winner, possibly after an initial transient. Leveraging the DB, we propose a plug-and-play \emph{reward calibration} (RC) that adaptively rebalances chosen versus rejected updates to satisfy the DB and mitigate likelihood displacement, without redesigning the base objective. Empirical results show that RC steers training toward more disentangled dynamics and often improves downstream performance across a range of objectives. Our code is available at https://github.com/IceyWuu/DisentangledPreferenceOptimization.

Related papers

• Preference Data Selection for Mitigating the Alignment Tax in Large Language Models [52.18310687523229]
  BALIGN is a balanced data selection strategy that mitigates catastrophic forgetting while optimizing alignment efficacy.<n>By aggregating three key data-centric features into a unified composite risk score, BALIGN systematically filters out high-risk preference samples.<n>Experiments on standard human preference datasets demonstrate that BALIGN strongly preserves foundational capabilities without compromising alignment gains.
  arXiv  Detail & Related papers  (2026-08-25T07:58:42Z)
• Unbiased Alignment for Large Language Models with Noisy Preferences [74.69408994904965]
  We present a theoretical framework for unbiased alignment, introducing the Unbiased Reward Model (URM) loss and the Unbiased Direct Preference Optimization (UDPO) loss.<n>By mathematically correcting the distortion induced by preference noise, our novel objectives enable unbiased model training directly from noisy datasets.
  arXiv  Detail & Related papers  (2026-07-03T12:04:43Z)
• Beyond Uniform Forgetting: A Study of Sequential Direct Preference Optimization Across Preference Settings [11.551054698858266]
  We study sequential Direct Preference optimisation across four preference settings.<n>We find that sequential DPO does not produce a single forgetting pattern.<n>Mechanistic diagnostics show that Stage2 gradients and adapter updates are near-orthogonal to the previous objective across all settings.
  arXiv  Detail & Related papers  (2026-06-18T03:20:41Z)
• DynamicPO: Dynamic Preference Optimization for Recommendation [25.125204801416587]
  In large language model (LLM)-based recommendation systems, direct preference optimization (DPO) effectively aligns recommendations with user preferences.<n>Our empirical analyses reveal a counterintuitive phenomenon, preference optimization collapse, where increasing the number of negative samples can lead to performance degradation.<n>We propose DynamicPO, a lightweight and plug-and-play framework comprising two adaptive mechanisms: Dynamic Boundary Negative Selection and Dual-Margin Dynamic beta Adjustment.
  arXiv  Detail & Related papers  (2026-05-01T01:06:38Z)
• ACPO: Counteracting Likelihood Displacement in Vision-Language Alignment with Asymmetric Constraints [26.309376587678354]
  We propose Asymmetric Constrained Preference Optimization (ACPO)<n>ACPO applies dynamic, target-oriented scaling to preference optimization.<n>We show that ACPO effectively reverses the chosen-reward degradation of standard DPO.
  arXiv  Detail & Related papers  (2026-03-23T16:26:11Z)
• From Noisy Traces to Stable Gradients: Bias-Variance Optimized Preference Optimization for Aligning Large Reasoning Models [90.45197506653341]
  Large reasoning models generate intermediate reasoning traces before producing final answers.<n> aligning LRMs with human preferences, a crucial prerequisite for model deployment, remains underexplored.<n>A common workaround optimized a single sampled trajectory, which introduces substantial gradient variance from trace sampling.
  arXiv  Detail & Related papers  (2025-10-06T17:58:01Z)
• Adaptive Helpfulness-Harmlessness Alignment with Preference Vectors [13.630818884973127]
  We propose Preference Vector, a novel framework inspired by task arithmetic.<n>Instead of optimizing multiple preferences within a single objective, we train separate models on individual preferences, extract behavior shifts as preference vectors, and dynamically merge them at test time.<n> Experiments show that our proposed Preference Vector framework improves helpfulness without excessive conservatism, allows smooth control over preference trade-offs, and supports scalable multi-preference alignment.
  arXiv  Detail & Related papers  (2025-04-27T12:16:51Z)
• Leveraging Robust Optimization for LLM Alignment under Distribution Shifts [51.74394601039711]
  Preference alignment methods are increasingly critical for steering large language models to generate outputs consistent with human values.<n>We propose a novel distribution-aware optimization framework that improves preference alignment despite such shifts.
  arXiv  Detail & Related papers  (2025-04-08T09:14:38Z)
• Few-shot Steerable Alignment: Adapting Rewards and LLM Policies with Neural Processes [50.544186914115045]
  Large language models (LLMs) are increasingly embedded in everyday applications.<n> Ensuring their alignment with the diverse preferences of individual users has become a critical challenge.<n>We present a novel framework for few-shot steerable alignment.
  arXiv  Detail & Related papers  (2024-12-18T16:14:59Z)
• Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization [60.176008034221404]
  Direct Preference Optimization (DPO) and its variants are increasingly used for aligning language models with human preferences.<n>Prior work has observed that the likelihood of preferred responses often decreases during training.<n>We demonstrate that likelihood displacement can be catastrophic, shifting probability mass from preferred responses to responses with an opposite meaning.
  arXiv  Detail & Related papers  (2024-10-11T14:22:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.