Rewards as Labels: Revisiting RLVR from a Classification Perspective

• URL: http://arxiv.org/abs/2602.05630v1
• Date: Thu, 05 Feb 2026 13:11:36 GMT
• Title: Rewards as Labels: Revisiting RLVR from a Classification Perspective
• Abstract summary: Reinforcement Learning with Verifiable Rewards has recently advanced the capabilities of Large Language Models in complex reasoning tasks.<n>We propose Rewards as Labels (REAL), a novel framework that revisits verifiable rewards as categorical labels rather than scalar weights.<n>Our analysis reveals that REAL induces a monotonic and bounded gradient weighting, enabling balanced gradient allocation across rollouts.
• Score: 10.774056916074441
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement Learning with Verifiable Rewards has recently advanced the capabilities of Large Language Models in complex reasoning tasks by providing explicit rule-based supervision. Among RLVR methods, GRPO and its variants have achieved strong empirical performance. Despite their success, we identify that they suffer from Gradient Misassignment in Positives and Gradient Domination in Negatives, which lead to inefficient and suboptimal policy updates. To address these issues, we propose Rewards as Labels (REAL), a novel framework that revisits verifiable rewards as categorical labels rather than scalar weights, thereby reformulating policy optimization as a classification problem. Building on this, we further introduce anchor logits to enhance policy learning. Our analysis reveals that REAL induces a monotonic and bounded gradient weighting, enabling balanced gradient allocation across rollouts and effectively mitigating the identified mismatches. Extensive experiments on mathematical reasoning benchmarks show that REAL improves training stability and consistently outperforms GRPO and strong variants such as DAPO. On the 1.5B model, REAL improves average Pass@1 over DAPO by 6.7%. These gains further scale to 7B model, REAL continues to outperform DAPO and GSPO by 6.2% and 1.7%, respectively. Notably, even with a vanilla binary cross-entropy, REAL remains stable and exceeds DAPO by 4.5% on average.

Related papers

• VeriGate: Verifier-Gated Step-Level Supervision for GRPO [51.26100506256885]
  Group Relative Policy Optimization is an effective recipe for training reasoning models with verifier-based outcome rewards.<n>We present VeriGate, a verifier-gated extension of GRPO that addresses these limitations with three design choices.<n>We show that VeriGate improves average accuracy by about 20% and 12% for 1.5B and 7B models respectively.
  arXiv  Detail & Related papers  (2026-05-28T18:20:32Z)
• Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigation [13.272542054938258]
  We introduce the Advantage Collapse Rate (ACR), the first metric quantifying the proportion of training batches with ineffective gradients.<n>We then propose Adaptive Virtual Sample Policy Optimization (AVSPO), a lightweight extension of GRPO that injects virtual reward samples.<n>AVSPO reduces advantage collapse by 58-63% relative to GRPO and yields consistent accuracy gains of 4-6 percentage points across all model scales.
  arXiv  Detail & Related papers  (2026-05-20T12:57:37Z)
• Learning from Failures: Correction-Oriented Policy Optimization with Verifiable Rewards [73.44333771806282]
  Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as an effective paradigm for improving the reasoning capabilities of large language models.<n>We propose Correction-Oriented Policy Optimization (CIPO), a simple and effective extension to RLVR that converts on-policy failed trajectories into correction-oriented supervision.<n>CIPO improves learning effectiveness while explicitly enhancing the model's ability to correct its own errors.
  arXiv  Detail & Related papers  (2026-05-14T08:22:21Z)
• Selector-Guided Autonomous Curriculum for One-Shot Reinforcement Learning from Verifiable Rewards [0.0]
  Reinforcement Learning from Verifiable Rewards (RLVR) has been established as a highly effective technique for augmenting the math reasoning skills of Large Language Models (LLMs)<n>We propose a Selector-Guided Autonomous Curriculum (SGAC) approach, which employs a learnable selector model on a multi-dimensional feature space consisting of success probability, reward variance, output disagreementtropy, and semantic difficulty level, instead of the static reward variance.<n>Our framework is evaluated using the Hendrycks MATH benchmark, with the Qwen2.5-Math-1.5B model serving as the baseline.
  arXiv  Detail & Related papers  (2026-05-03T11:10:01Z)
• Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing [79.88256756334327]
  Self-Distillation Policy Optimization (SDPO) addresses this by providing denser, more targeted logit-level supervision.<n>We propose Sample-Routed Policy Optimization (SRPO)<n>SRPO routes correct samples to GRPO's reward-aligned reinforcement and failed samples to SDPO's targeted logit-level correction.
  arXiv  Detail & Related papers  (2026-04-02T17:29:18Z)
• Back to Basics: Revisiting Exploration in Reinforcement Learning for LLM Reasoning via Generative Probabilities [10.235183326885794]
  Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as an indispensable paradigm for enhancing reasoning in Large Language Models (LLMs)<n>We analyze this issue from the perspective of sampling probability dynamics, identifying that the standard objective disproportionately reinforces the highest-likelihood paths.<n>We propose a novel Advantage Re-weighting Mechanism (ARM) designed to equilibrate the confidence levels across all correct responses.
  arXiv  Detail & Related papers  (2026-02-05T04:06:55Z)
• CARE What Fails: Contrastive Anchored-REflection for Verifiable Multimodal [84.71254539482369]
  Group-relative reinforcement learning with verifiable rewards (RLVR) often wastes the most informative data it already has the failures.<n>We present CARE, a failure-centric post-training framework for multimodal reasoning that turns errors into supervision.<n> CARE improves accuracy and training smoothness while explicitly increasing the share of learning signal that comes from failures.
  arXiv  Detail & Related papers  (2025-12-22T16:34:21Z)
• DaGRPO: Rectifying Gradient Conflict in Reasoning via Distinctiveness-Aware Group Relative Policy Optimization [20.66452395111739]
  We propose Distinctiveness-aware Group Relative Policy Optimization (DaGRPO)<n>DaGRPO incorporates two core mechanisms: (1) Sequence-level Gradient Rectification, which utilizes fine-grained scoring to dynamically mask sample pairs with low distinctiveness; and (2) Off-policy Data Augmentation, which introduces high-quality anchors to recover training signals for challenging tasks.<n>In-depth analysis confirms that DaGRPO effectively mitigates gradient explosion and accelerates the emergence of long-chain reasoning capabilities.
  arXiv  Detail & Related papers  (2025-12-06T07:51:36Z)
• DARO: Difficulty-Aware Reweighting Policy Optimization [18.07946696398167]
  Group Relative Policy Optimization ( GRPO) has emerged as the de facto approach for Reinforcement Learning with Verifiable Rewards (RLVR)<n>We provide a unified view, demonstrating that their reliance on static or overly simplistic weighting schemes tied to sample difficulty prevents adaptation to a model's evolving capabilities.<n>We introduce bfbfDifficulty-Aware Reweighting Policy Optimization (DARO), a method that dynamically adjusts the loss contribution of each difficulty group based on the model's learning state.
  arXiv  Detail & Related papers  (2025-10-10T04:57:15Z)
• SPARK: Synergistic Policy And Reward Co-Evolving Framework [84.22494672256894]
  We introduce the Synergistic Policy And Reward Co-Evolving Framework (SPARK), an efficient, on-policy, and stable method that builds on RLVR.<n>Instead of discarding rollouts and correctness data, SPARK recycles this valuable information to simultaneously train the model itself as a generative reward model.<n>We show that SPARK achieves significant performance gains on multiple LLM and LVLM models and multiple reasoning, reward models, and general benchmarks.
  arXiv  Detail & Related papers  (2025-09-26T17:50:12Z)
• LLaVA-Critic-R1: Your Critic Model is Secretly a Strong Policy Model [99.71684530652942]
  We show that LLaVA-Critic-R1 emerges as a top-performing critic but also as a competitive policy model.<n>Applying self-critique at test time yields an average +13.8% improvement on five representative reasoning tasks.<n>Our results reveal that RL training on critic data can produce a unified model excelling at both evaluation and generation.
  arXiv  Detail & Related papers  (2025-08-31T03:08:02Z)
• GRPO-CARE: Consistency-Aware Reinforcement Learning for Multimodal Reasoning [53.894789613838654]
  We introduce SEED-Bench-R1, a benchmark with complex real-world videos requiring balanced perception and reasoning.<n>Using SEED-Bench-R1, we find that standard GRPO, while improving answer accuracy, often reduces logical coherence between reasoning steps and answers, with only a 57.9% consistency rate.<n>We propose GRPO-CARE, a consistency-aware RL framework optimizing both answer correctness and reasoning coherence without explicit supervision.
  arXiv  Detail & Related papers  (2025-06-19T08:49:13Z)
• REARANK: Reasoning Re-ranking Agent via Reinforcement Learning [69.8397511935806]
  We present REARANK, a large language model (LLM)-based listwise reasoning reranking agent.<n>REARANK explicitly reasons before reranking, significantly improving both performance and interpretability.
  arXiv  Detail & Related papers  (2025-05-26T14:31:48Z)
• DisCO: Reinforcing Large Reasoning Models with Discriminative Constrained Optimization [50.91849555841057]
  Group Relative Policy Optimization is a reinforcement learning method for large reasoning models (LRMs)<n>We introduce a new Discriminative Constrained Optimization framework for reinforcing LRMs, grounded in the principle of discriminative learning.<n>DisCO significantly outperforms GRPO and its improved variants such as DAPO, achieving average gains of 7% over GRPO and 6% over DAPO.
  arXiv  Detail & Related papers  (2025-05-18T11:08:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.