S-SPPO: Semantic-Calibrated Self-Play Preference Optimization

• URL: http://arxiv.org/abs/2606.01561v1
• Date: Mon, 01 Jun 2026 02:06:58 GMT
• Title: S-SPPO: Semantic-Calibrated Self-Play Preference Optimization
• Abstract summary: We introduce Self-Play Preference Optimization (SPPO), which iteratively refines the policy by training on self-generated win-lose pairs.<n>Our investigation reveals a critical instability in SPPO: the optimization is prone to policy degeneration when the preference assigns overly confident wins to semantically indistinguishable responses.<n>We show that S-SPPO avoids the performance degradation seen in prior methods, achieving 52.19% win rate and 47.46% length-controlled win rate on AlpacaEval 2.0 with Llama-3-8B, without using additional human-annotated preferences during training.
• Score: 36.01916066772865
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Aligning Large Language Models (LLMs) with human preferences is often formulated via Direct Preference Optimization (DPO). However, the standard Bradley-Terry instantiation of DPO is limited in modeling common departures from transitivity in human preferences. To address this, recent work has introduced Self-Play Preference Optimization (SPPO), which iteratively refines the policy by training on self-generated win-lose pairs. Our investigation, however, reveals a critical instability in SPPO: the optimization is prone to policy degeneration when the preference oracle assigns overly confident wins to semantically indistinguishable responses. To mitigate this, we propose S-SPPO, a dual-space semantic calibration framework comprising: i) Supervision Calibration via semantic gating, which anneals win rate targets toward the maximum-entropy baseline as semantic overlap increases; and ii) Representation Calibration via latent repulsion to enforce geometric diversity to prevent manifold collapse and maintain latent diversity between chosen and rejected samples. Theoretically, we show that the calibration preserves the constant-sum game structure, facilitating convergence to a Nash Equilibrium. Empirically, S-SPPO avoids the performance degradation seen in prior methods, achieving 52.19% win rate and 47.46% length-controlled win rate on AlpacaEval 2.0 with Llama-3-8B, without using additional human-annotated preferences during training. The code will be available at https://github.com/xiwenc1/s-sppo.

Related papers

• Boosting Direct Preference Optimization with Penalization [0.0]
  Direct Preference Optimization with Penalization (DPOP) is a simple extension of DPO that augments the base preference loss with a gated penalty on reference-greedy responses.<n>On AlpacaEval 2.0, DPOP improves length-controlled win rate over DPO, SimPO, and AlphaDPO on both Llama-3-8b-it and Gemma-2-9b-it.
  arXiv  Detail & Related papers  (2026-06-10T17:22:51Z)
• AdaDPO: Self-Adaptive Direct Preference Optimization with Balanced Gradient Updates [0.03999851878220877]
  We propose a Self-Adaptive variant of the DPO algorithm that introduces per-preference-pair, stop-gradient-based coefficients.<n>AdaDPO is constructed to enforce equality of gradient magnitudes between preferred and dispreferred probabilities.<n>Because it operates purely at the loss level, AdaDPO can be dropped into existing preference-based alignment pipelines.
  arXiv  Detail & Related papers  (2026-05-27T13:05:49Z)
• Conditional Equivalence of DPO and RLHF: Implicit Assumption, Failure Modes, and Provable Alignment [51.18269946911088]
  Direct Preference Optimization (DPO) has emerged as a popular alternative to Reinforcement Learning from Human Feedback (RLHF)<n>We show that DPO and RLHF optimize fundamentally different objectives in such cases.<n>We introduce Constrained Preference Optimization (CPO), augmenting RLHF with constraints for provable alignment.<n>Our theoretical analysis establishes when DPOs' guarantees hold and provides solutions preserving simplicity with provable alignment.
  arXiv  Detail & Related papers  (2026-05-20T07:26:22Z)
• ACPO: Counteracting Likelihood Displacement in Vision-Language Alignment with Asymmetric Constraints [26.309376587678354]
  We propose Asymmetric Constrained Preference Optimization (ACPO)<n>ACPO applies dynamic, target-oriented scaling to preference optimization.<n>We show that ACPO effectively reverses the chosen-reward degradation of standard DPO.
  arXiv  Detail & Related papers  (2026-03-23T16:26:11Z)
• wDPO: Winsorized Direct Preference Optimization for Robust LLM Alignment [48.487557157323664]
  In practice, preference data are often noisy.<n>Existing robust variants of DPO mainly rely on uniform objective modifications or global reweighting.<n>We show that robust preference alignment benefits from addressing different noise types with targeted interventions.
  arXiv  Detail & Related papers  (2026-03-07T13:30:53Z)
• Anchoring Values in Temporal and Group Dimensions for Flow Matching Model Alignment [61.80228667422234]
  VGPO redefines value estimation across both temporal and group dimensions.<n>It transforms the sparse terminal reward into dense, process-aware value estimates.<n>It replaces standard group normalization with a novel process enhanced by absolute values to maintain a stable optimization signal.
  arXiv  Detail & Related papers  (2025-12-13T16:31:26Z)
• Uncertainty-Penalized Direct Preference Optimization [52.387088396044206]
  We develop a pessimistic framework for DPO by introducing preference uncertainty penalization schemes. The penalization serves as a correction to the loss which attenuates the loss gradient for uncertain samples. We show improved overall performance compared to vanilla DPO, as well as better completions on prompts from high-uncertainty chosen/rejected responses.
  arXiv  Detail & Related papers  (2024-10-26T14:24:37Z)
• Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer [52.09480867526656]
  We identify the source of misalignment as a form of distributional shift and uncertainty in learning human preferences.<n>To mitigate overoptimization, we first propose a theoretical algorithm that chooses the best policy for an adversarially chosen reward model.<n>Using the equivalence between reward models and the corresponding optimal policy, the algorithm features a simple objective that combines a preference optimization loss and a supervised learning loss.
  arXiv  Detail & Related papers  (2024-05-26T05:38:50Z)
• Self-Play Preference Optimization for Language Model Alignment [75.83359213697854]
  Recent advancements suggest that directly working with preference probabilities can yield a more accurate reflection of human preferences. We propose a self-play-based method for language model alignment, which treats the problem as a constant-sum two-player game. Our approach, dubbed Self-Play Preference Optimization (SPPO), utilizes iterative policy updates to provably approximate the Nash equilibrium.
  arXiv  Detail & Related papers  (2024-05-01T17:59:20Z)
• Statistical Rejection Sampling Improves Preference Optimization [42.57245965632205]
  We introduce a novel approach to source preference data from the target optimal policy using rejection sampling. We also propose a unified framework that enhances the loss functions used in both Sequence Likelihood (SLiC) and Direct Preference Optimization (DPO) from a preference modeling standpoint.
  arXiv  Detail & Related papers  (2023-09-13T01:07:25Z)
• You May Not Need Ratio Clipping in PPO [117.03368180633463]
  Proximal Policy Optimization (PPO) methods learn a policy by iteratively performing multiple mini-batch optimization epochs of a surrogate objective with one set of sampled data. Ratio clipping PPO is a popular variant that clips the probability ratios between the target policy and the policy used to collect samples. We show in this paper that such ratio clipping may not be a good option as it can fail to effectively bound the ratios. We show that ESPO can be easily scaled up to distributed training with many workers, delivering strong performance as well.
  arXiv  Detail & Related papers  (2022-01-31T20:26:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.