A Unified Pair-GRPO Family: From Implicit to Explicit Preference Constraints for Stable and General RL Alignment

• URL: http://arxiv.org/abs/2605.06375v1
• Date: Thu, 07 May 2026 14:56:11 GMT
• Title: A Unified Pair-GRPO Family: From Implicit to Explicit Preference Constraints for Stable and General RL Alignment
• Abstract summary: Large language model (LLM) alignment via preference learning suffers from unstable policy updates, ambiguous gradient directions, and high gradient variance.<n>We establish a unified theoretical framework for preference-based RL optimization centered on the Pair-GRPO family.<n>We provide comprehensive theoretical guarantees for both variants--including monotonic policy improvement, deterministic gradient direction, gradient-variance reduction, and dynamic step-size convergence.
• Score: 6.3333468199326255
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language model (LLM) alignment via reinforcement learning from human preferences (RLHF) suffers from unstable policy updates, ambiguous gradient directions, poor interpretability, and high gradient variance in mainstream pairwise preference learning paradigms. To systematically address these limitations, we establish a unified theoretical framework for preference-based RL optimization centered on the Pair-GRPO family, comprising two tightly coupled variants: Soft-Pair-GRPO and Hard-Pair-GRPO. Soft-Pair-GRPO is a minimal modification of Group Relative Policy Optimization (GRPO) that replaces group-normalized scalar rewards with binary pairwise preference rewards, retaining GRPO's clipped surrogate and KL-regularized structure. We prove a critical gradient equivalence theorem: under first-order Taylor expansion around the current policy, Soft-Pair-GRPO's gradient is a positive scalar multiple of standard GRPO's gradient, explaining its empirical stability despite discarding continuous reward magnitudes. Building on this foundation, we propose Hard-Pair-GRPO, an advanced variant introducing explicit local probability constraints and constrained KL-fitting optimization to further suppress gradient noise and global policy drift. We provide comprehensive theoretical guarantees for both variants--including monotonic policy improvement, deterministic gradient direction, gradient-variance reduction, and dynamic step-size convergence. Extensive experiments on standard LLM alignment benchmarks (HH-RLHF,UltraFeedback) and the MuJoCo continuous control task HalfCheetah-v4 demonstrate that our Pair-GRPO family consistently outperforms state-of-the-art baselines in alignment quality, human preference win rate, training stability, and generalization to general reinforcement learning. Ablation studies validate the critical contributions of each core component.

Related papers

• GUPO: Gradient Uncertainty-aware Policy Optimization for Post-Training Large Language Models [44.63057841001313]
  Group Relative Policy Optimization (GRPO) has become a widely used approach for post-training Large Language Models (LLMs)<n>In GRPO, the group gradients induced by different queries within the same mini-batch are directly averaged to form the policy update.<n>Our empirical analysis suggests that group-gradient conflicts tend to be associated with less effective policy updates.
  arXiv  Detail & Related papers  (2026-08-18T06:22:55Z)
• MDP-GRPO: Stabilized Group Relative Policy Optimization for Multi-Constraint Instruction Following [4.563830993050022]
  Reinforcement learning with verifiable rewards is ideal for multi-constraint instruction following.<n>Standard group-relative policy optimization becomes unstable under discrete, low-dispersion rewards.<n>We propose MDP-GRPO, which stabilizes learning through multi-temperature sampling to increase reward dispersion.
  arXiv  Detail & Related papers  (2026-06-04T11:58:59Z)
• Holder Policy Optimisation [26.521180498291717]
  textbfHlderPO is a generalised policy optimisation framework.<n>It unifies token-level probability aggregation via the Hlder mean.<n>It achieves a state-of-the-art average accuracy of $54.9%$ across multiple mathematical benchmarks.
  arXiv  Detail & Related papers  (2026-05-12T12:45:03Z)
• Too Correct to Learn: Reinforcement Learning on Saturated Reasoning Data [55.84428098924793]
  We propose Constrained Uniform Top-K Sampling (CUTS), a parameter-free decoding strategy enforcing structure-preserving exploration.<n>We integrate this into Mixed-CUTS, a training framework synergizing exploitative and exploratory rollouts to amplify intra-group advantage variance.<n> Notably, Mixed-CUTS improves Pass@1 accuracy on the challenging AIME25 benchmark by up to 15.1% over standard GRPO.
  arXiv  Detail & Related papers  (2026-04-20T16:43:28Z)
• Personalized Group Relative Policy Optimization for Heterogenous Preference Alignment [13.085014101959118]
  We introduce Personalized GRPO, a novel alignment framework that decouples advantage estimation from batch statistics.<n>We evaluate P-GRPO across diverse tasks and find that it consistently achieves faster convergence and higher rewards than standard GRPO.
  arXiv  Detail & Related papers  (2026-02-17T19:00:43Z)
• f-GRPO and Beyond: Divergence-Based Reinforcement Learning Algorithms for General LLM Alignment [15.396104072574104]
  We propose f-Group Relative Policy Optimization (f-GRPO), a class of on-policy reinforcement learning, and f-Hybrid Alignment Loss (f-HAL)<n>We provide theoretical guarantees that these classes of objectives improve the average reward after alignment.
  arXiv  Detail & Related papers  (2026-02-05T18:01:52Z)
• Rethinking the Trust Region in LLM Reinforcement Learning [72.25890308541334]
  Proximal Policy Optimization (PPO) serves as the de facto standard algorithm for Large Language Models (LLMs)<n>We propose Divergence Proximal Policy Optimization (DPPO), which substitutes clipping with a more principled constraint.<n>DPPO achieves superior training and efficiency compared to existing methods, offering a more robust foundation for RL-based fine-tuning.
  arXiv  Detail & Related papers  (2026-02-04T18:59:04Z)
• MAESTRO: Meta-learning Adaptive Estimation of Scalarization Trade-offs for Reward Optimization [56.074760766965085]
  Group-Relative Policy Optimization has emerged as an efficient paradigm for aligning Large Language Models (LLMs)<n>We propose MAESTRO, which treats reward scalarization as a dynamic latent policy, leveraging the model's terminal hidden states as a semantic bottleneck.<n>We formulate this as a contextual bandit problem within a bi-level optimization framework, where a lightweight Conductor network co-evolves with the policy by utilizing group-relative advantages as a meta-reward signal.
  arXiv  Detail & Related papers  (2026-01-12T05:02:48Z)
• GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization [133.27496265096445]
  We show how to apply Group Relative Policy Optimization under multi-reward setting without examining its suitability.<n>We then introduce Group reward-Decoupled Normalization Policy Optimization (GDPO), a new policy optimization method to resolve these issues.<n>GDPO consistently outperforms GRPO, demonstrating its effectiveness and generalizability for multi-reward reinforcement learning optimization.
  arXiv  Detail & Related papers  (2026-01-08T18:59:24Z)
• Anchoring Values in Temporal and Group Dimensions for Flow Matching Model Alignment [61.80228667422234]
  VGPO redefines value estimation across both temporal and group dimensions.<n>It transforms the sparse terminal reward into dense, process-aware value estimates.<n>It replaces standard group normalization with a novel process enhanced by absolute values to maintain a stable optimization signal.
  arXiv  Detail & Related papers  (2025-12-13T16:31:26Z)
• On Global Convergence Rates for Federated Policy Gradient under Heterogeneous Environment [14.366821866598803]
  We introduce b-RS-FedPG, a novel policy gradient method that employs a carefully constructed softmax-inspired parameterization.<n>We demonstrate explicit convergence rates for b-RS-FedPG toward near-optimal stationary policies.
  arXiv  Detail & Related papers  (2025-05-29T14:08:35Z)
• DISCO Balances the Scales: Adaptive Domain- and Difficulty-Aware Reinforcement Learning on Imbalanced Data [65.09939942413651]
  We propose a principled extension to GRPO that addresses inter-group imbalance with two key innovations.<n> Domain-aware reward scaling counteracts frequency bias by reweighting optimization based on domain prevalence.<n>Difficulty-aware reward scaling leverages prompt-level self-consistency to identify and prioritize uncertain prompts that offer greater learning value.
  arXiv  Detail & Related papers  (2025-05-21T03:43:29Z)
• GVPO: Group Variance Policy Optimization for Large Language Model Post-Training [19.005045649097987]
  Group Variance Policy Optimization (GVPO) incorporates the analytical solution to KL-constrained reward directly into its weights.<n>GVPO offers two key advantages: it guarantees a unique optimal solution, exactly the KL-constrained reward objective, and it supports flexible sampling distributions.<n>By unifying theoretical guarantees with practical adaptability, GVPO establishes a new paradigm for reliable and versatile LLM post-training.
  arXiv  Detail & Related papers  (2025-04-28T09:02:24Z)
• PARL: A Unified Framework for Policy Alignment in Reinforcement Learning from Human Feedback [106.63518036538163]
  We present a novel unified bilevel optimization-based framework, textsfPARL, formulated to address the recently highlighted critical issue of policy alignment in reinforcement learning. Our framework addressed these concerns by explicitly parameterizing the distribution of the upper alignment objective (reward design) by the lower optimal variable. Our empirical results substantiate that the proposed textsfPARL can address the alignment concerns in RL by showing significant improvements.
  arXiv  Detail & Related papers  (2023-08-03T18:03:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.