Beyond Entropy: Correctness-Aware Advantage Shaping via Contrastive Policy Optimization

• URL: http://arxiv.org/abs/2607.14614v1
• Date: Thu, 16 Jul 2026 06:25:19 GMT
• Title: Beyond Entropy: Correctness-Aware Advantage Shaping via Contrastive Policy Optimization
• Abstract summary: entropy cannot distinguish useful uncertainty from detrimental confusion.<n>We propose Contrastive Policy Optimization (CPO), which uses token-level contrastive disagreement for correctness-aware advantage shaping.<n>CPO substantially outperforms entropy-based RLVR methods while maintaining strong generalization.
• Score: 35.632335113369116
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement learning with verifiable rewards (RLVR) commonly uses entropy for advantage shaping. However, entropy cannot distinguish useful uncertainty from detrimental confusion, limiting its effectiveness as a correctness signal. We propose Contrastive Policy Optimization (CPO), which uses token-level contrastive disagreement between reference-guided and vanilla generation distributions for correctness-aware advantage shaping. Both theoretical and empirical results show that this disagreement reliably indicates token-level correctness. We further show that On-policy Distillation is a special case of CPO, where the posterior distribution is instantiated by an external teacher model. CPO also resolves the zero-advantage problem. Experiments on in-domain and out-of-domain benchmarks demonstrate that CPO substantially outperforms entropy-based RLVR methods while maintaining strong generalization. Further analysis shows that correct and incorrect responses naturally support exploration and exploitation respectively, and balancing both leads to the best performance.

Related papers

• Stabilizing On-Policy Distillation for MLLM Reasoning with Global Normalization [9.860371691501076]
  On-policy distillation (OPD) has recently emerged as an important post-training paradigm.<n>We propose Globally Normalized Distillation Policy Optimization (GNDPO), a practical method that stabilizes optimization by transforming raw KL scores into batch-level relative advantages.
  arXiv  Detail & Related papers  (2026-06-08T06:41:31Z)
• Conditional Equivalence of DPO and RLHF: Implicit Assumption, Failure Modes, and Provable Alignment [51.18269946911088]
  Direct Preference Optimization (DPO) has emerged as a popular alternative to Reinforcement Learning from Human Feedback (RLHF)<n>We show that DPO and RLHF optimize fundamentally different objectives in such cases.<n>We introduce Constrained Preference Optimization (CPO), augmenting RLHF with constraints for provable alignment.<n>Our theoretical analysis establishes when DPOs' guarantees hold and provides solutions preserving simplicity with provable alignment.
  arXiv  Detail & Related papers  (2026-05-20T07:26:22Z)
• Learning from Failures: Correction-Oriented Policy Optimization with Verifiable Rewards [73.44333771806282]
  Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as an effective paradigm for improving the reasoning capabilities of large language models.<n>We propose Correction-Oriented Policy Optimization (CIPO), a simple and effective extension to RLVR that converts on-policy failed trajectories into correction-oriented supervision.<n>CIPO improves learning effectiveness while explicitly enhancing the model's ability to correct its own errors.
  arXiv  Detail & Related papers  (2026-05-14T08:22:21Z)
• Entropy Polarity in Reinforcement Fine-Tuning: Direction, Asymmetry, and Control [77.8471519867791]
  Empirically, we show that entropy polarity reliably predicts entropy changes.<n>We propose Polarity-Aware Policy Optimization (PAPO), which preserves both polarity branches and implements entropy control through advantage reweighting.
  arXiv  Detail & Related papers  (2026-05-12T08:47:05Z)
• Taming Extreme Tokens: Covariance-Aware GRPO with Gaussian-Kernel Advantage Reweighting [35.85110049085719]
  Group Relative Policy Optimization ( GRPO) has emerged as a promising approach for improving the reasoning capabilities of large language models.<n>We propose a hyper parameter-free, covariance-weighted optimization method that dynamically down-weights extreme token-level updates.<n>Our approach improves downstream performance across reasoning benchmarks compared with GRPO, and effectively stablizes entropy as training progresses.
  arXiv  Detail & Related papers  (2026-05-12T05:05:47Z)
• Know What You Know: Metacognitive Entropy Calibration for Verifiable RL Reasoning [31.629261193485053]
  Large reasoning models (LRMs) have emerged as a powerful paradigm for solving complex real-world tasks.<n>Most existing outcome-only RLVR pipelines rely almost exclusively on a binary correctness signal and largely ignore the model's intrinsic uncertainty.<n>We propose EGPO, a metacognitive entropy calibration framework that explicitly integrates intrinsic uncertainty into RLVR for enhancing LRMs.
  arXiv  Detail & Related papers  (2026-02-26T08:40:06Z)
• Uncalibrated Reasoning: GRPO Induces Overconfidence for Stochastic Outcomes [55.2480439325792]
  Reinforcement learning (RL) has proven remarkably effective at improving the accuracy of language models in verifiable and deterministic domains like mathematics.<n>Here, we examine if current RL methods are also effective at optimizing language models in verifiable domains with outcomes, like scientific experiments.
  arXiv  Detail & Related papers  (2025-08-15T20:50:53Z)
• The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models [99.98293908799731]
  This paper aims to overcome a major obstacle in scaling RL for reasoning with LLMs, namely the collapse of policy entropy.<n>In practice, we establish a transformation equation R=-a*eH+b between entropy H and downstream performance R.<n>We propose two simple yet effective techniques, namely Clip-Cov and KL-Cov, which clip and apply KL penalty to tokens with high covariances respectively.
  arXiv  Detail & Related papers  (2025-05-28T17:38:45Z)
• Mirror Descent Actor Critic via Bounded Advantage Learning [0.0]
  Mirror Descent Value Iteration (MDVI) uses both Kullback-Leibler divergence and entropy as regularizers in its value and policy updates.<n>We propose Mirror Descent Actor Critic (MDAC) as an actor-critic style instantiation of MDVI for continuous action domains.
  arXiv  Detail & Related papers  (2025-02-06T08:14:03Z)
• Fine-Tuning Language Models with Advantage-Induced Policy Alignment [80.96507425217472]
  We propose a novel algorithm for aligning large language models to human preferences. We show that it consistently outperforms PPO in language tasks by a large margin. We also provide a theoretical justification supporting the design of our loss function.
  arXiv  Detail & Related papers  (2023-06-04T01:59:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.