Don't Peek at the Answer: Outcome-Masked Group Relative Policy Optimization for Label-Free RLVR

• URL: http://arxiv.org/abs/2608.03119v1
• Date: Tue, 04 Aug 2026 04:41:20 GMT
• Title: Don't Peek at the Answer: Outcome-Masked Group Relative Policy Optimization for Label-Free RLVR
• Abstract summary: Voting-based label-free RLVR replace gold supervision with answer-level consensus from model samples.<n>We propose OM-GRPO, a label-free RLVR framework that decouples reward estimation from policy optimization.
• Score: 21.781083492293913
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) improves LLM reasoning but typically relies on ground-truth (GT) answers, limiting scalability. Voting-based label-free RLVR replace gold supervision with answer-level consensus from model samples. However, collapse arises when the same answer-level signal is used both to estimate rewards and to drive token-level policy optimization, encouraging the model to directly reinforce answer tokens rather than improve reasoning. We propose OM-GRPO, a label-free RLVR framework that decouples reward estimation from policy optimization. OM-GRPO masks gradients on the answer span while retaining answer-level rewards through a soft consensus signal, shifting optimization pressure away from answer tokens. We further introduce Contrast-Augmented Reward, which refines reward estimation via low-cost pairwise comparisons over existing trajectories without additional rollouts. Across diverse reasoning benchmarks and three LLM backbones, OM-GRPO consistently outperforms existing label-free RLVR methods and matches supervised GT-reward training with stable optimization. This stability is particularly beneficial in the Test-Time Training setting, where OM-GRPO surpasses majority voting by 4.24 points.

Related papers

• On-policy Distillation with Verifiable Reward [32.72049308672042]
  On-policy distillation (OPD) andReinforcement Learning with Verifiable Rewards (RLVR) have become two widely adopted paradigms for post-training large language models.<n>We propose On-policy Distillation with Verifiable Reward (OPDVR), a simple yet effective method that seamlessly combines OPD and RLVR.<n> Experiments on six reasoning benchmarks show that OPDVR consistently outperforms standard OPD.
  arXiv  Detail & Related papers  (2026-08-25T15:21:17Z)
• Two is better than one: A Collapse-free Multi-Reward RLIF Training Framework [6.490241400619907]
  Reinforcement learning from internal feedback has recently emerged as a scalable unsupervised alternative.<n>We propose a multi-reward RLIF framework that decomposes the training signal into two complementary components.<n>We show that our method can support stable long-horizon reasoning without relying on external ground-truth supervision.
  arXiv  Detail & Related papers  (2026-05-21T15:30:47Z)
• LamPO: A Lambda Style Policy Optimization for Reasoning Language Models [34.349722314481824]
  Reinforcement learning with verifiable rewards (RLVR) has become an effective paradigm for improving reasoning language models on tasks such as mathematics, coding, and scientific question answering.<n>We propose textbfLamPO, a textbfLambda-Style Policy Optimization method that replaces scalar group advantages with a emphPairwise De Advantage.<n>Experiments on AIME24, AIME25, MATH-500, and GPQA-Diamond with Qwen3-1.7B, Qwen3-4B, and Phi-4-mini show that LamPO consistently improves over GRPO
  arXiv  Detail & Related papers  (2026-05-20T14:24:11Z)
• Verifier-Free RL for LLMs via Intrinsic Gradient-Norm Reward [69.99652051809737]
  In this work, we propose Verifier-free Intrinsic Gradient-Norm Reward (VIGOR)<n>VIGOR is a simple reward that uses only the policy model itself.<n>It exhibits cross-domain transfer to code benchmarks when trained only on math data.
  arXiv  Detail & Related papers  (2026-05-11T03:15:37Z)
• JURY-RL: Votes Propose, Proofs Dispose for Label-Free RLVR [39.03968285406107]
  JURY-RL is a label-free RLVR framework that decouples answer proposal from reward disposal.<n>It consistently outperforms other label-free baselines on mathematical reasoning benchmarks.<n>It attains pass@1 performance comparable to supervised ground-truth training.
  arXiv  Detail & Related papers  (2026-04-28T09:29:00Z)
• CLIPO: Contrastive Learning in Policy Optimization Generalizes RLVR [10.821950260274066]
  Reinforcement Learning with Verifiable Rewards (RLVR) has significantly advanced the reasoning capacity of Large Language Models (LLMs)<n>Training on process-wrong but outcome-correct rollouts can lead to hallucination and answer-copying.<n>We incorporate a Contrastive Learning mechanism into the Policy Optimization (CLIPO) to generalize the RLVR process.
  arXiv  Detail & Related papers  (2026-03-10T17:59:54Z)
• SPARK: Synergistic Policy And Reward Co-Evolving Framework [84.22494672256894]
  We introduce the Synergistic Policy And Reward Co-Evolving Framework (SPARK), an efficient, on-policy, and stable method that builds on RLVR.<n>Instead of discarding rollouts and correctness data, SPARK recycles this valuable information to simultaneously train the model itself as a generative reward model.<n>We show that SPARK achieves significant performance gains on multiple LLM and LVLM models and multiple reasoning, reward models, and general benchmarks.
  arXiv  Detail & Related papers  (2025-09-26T17:50:12Z)
• Implicit Actor Critic Coupling via a Supervised Learning Framework for RLVR [25.56828724912418]
  Recent advances in Reinforcement Learning with Verifiable Rewards (RLVR) have empowered large language models (LLMs) to tackle challenging reasoning tasks such as mathematics and programming.<n>Despite its promise, the RLVR paradigm poses significant challenges, as existing methods often suffer from sparse reward signals and unstable policy gradient updates.<n>We propose $textbfPACS$, a novel RLVR framework that achieves im$textbfP$licit $textbfA$ctor $textbfC$ritic coupling.
  arXiv  Detail & Related papers  (2025-09-02T17:22:46Z)
• RLPR: Extrapolating RLVR to General Domains without Verifiers [103.14103272635893]
  We propose RLPR, a simple verifier-free framework that extrapolates RLVR to broader general domains.<n>We find that addressing the high variance of this noisy probability reward is crucial to make it work.<n>RLPR consistently improves reasoning capabilities in both areas for Gemma, Llama, and Qwen based models.
  arXiv  Detail & Related papers  (2025-06-23T02:56:36Z)
• GRPO-CARE: Consistency-Aware Reinforcement Learning for Multimodal Reasoning [53.894789613838654]
  We introduce SEED-Bench-R1, a benchmark with complex real-world videos requiring balanced perception and reasoning.<n>Using SEED-Bench-R1, we find that standard GRPO, while improving answer accuracy, often reduces logical coherence between reasoning steps and answers, with only a 57.9% consistency rate.<n>We propose GRPO-CARE, a consistency-aware RL framework optimizing both answer correctness and reasoning coherence without explicit supervision.
  arXiv  Detail & Related papers  (2025-06-19T08:49:13Z)
• Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning [55.15106182268834]
  Reinforcement learning with verifiable rewards (RLVR) has emerged as the leading approach for enhancing reasoning capabilities in large language models.<n>It faces a fundamental compute and memory asymmetry: rollout generation is embarrassingly parallel and memory-light, whereas policy updates are communication-heavy and memory-intensive.<n>We introduce PODS (Policy Optimization with Down-Sampling), which decouples rollout generation from policy updates by training only on a strategically selected subset of rollouts.
  arXiv  Detail & Related papers  (2025-04-18T17:49:55Z)
• REBEL: Reinforcement Learning via Regressing Relative Rewards [59.68420022466047]
  We propose REBEL, a minimalist RL algorithm for the era of generative models.<n>In theory, we prove that fundamental RL algorithms like Natural Policy Gradient can be seen as variants of REBEL.<n>We find that REBEL provides a unified approach to language modeling and image generation with stronger or similar performance as PPO and DPO.
  arXiv  Detail & Related papers  (2024-04-25T17:20:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.