DelTA: Discriminative Token Credit Assignment for Reinforcement Learning from Verifiable Rewards

• URL: http://arxiv.org/abs/2605.21467v1
• Date: Wed, 20 May 2026 17:53:09 GMT
• Title: DelTA: Discriminative Token Credit Assignment for Reinforcement Learning from Verifiable Rewards
• Abstract summary: Reinforcement learning from verifiable rewards (RLVR) has emerged as a central technique for improving the reasoning capabilities of large language models.<n>We introduce a discriminator view of RLVR updates, showing that the policy-gradient update direction implicitly acts as a linear discriminator over token-gradient vectors.<n>We propose DelTA, a discriminative token credit assignment method that estimates token coefficients to amplify side-specific token-gradient directions and downweight shared or weakly discriminative ones.
• Score: 41.76858591194076
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Reinforcement learning from verifiable rewards (RLVR) has emerged as a central technique for improving the reasoning capabilities of large language models. Despite its effectiveness, how response-level rewards translate into token-level probability changes remains poorly understood. We introduce a discriminator view of RLVR updates, showing that the policy-gradient update direction implicitly acts as a linear discriminator over token-gradient vectors and thereby determines which token probabilities are increased or decreased during learning. Under standard sequence-level RLVR, this discriminator is constructed from positive- and negative-side centroids formed by advantage-weighted averaging of token-gradient vectors. However, such centroid construction can be dominated by shared high-frequency patterns, such as formatting tokens, diluting sparse yet discriminative directions that better distinguish high-reward responses from low-reward ones. To address this limitation, we propose $\textbf{DelTA}$, a discriminative token credit assignment method that estimates token coefficients to amplify side-specific token-gradient directions and downweight shared or weakly discriminative ones. These coefficients reweight a self-normalized RLVR surrogate, making the effective side-wise centroids more contrastive and thereby reshaping the RLVR update direction. On seven mathematical benchmarks, DelTA outperforms the strongest same-scale baselines by 3.26 and 2.62 average points on Qwen3-8B-Base and Qwen3-14B-Base, respectively. Additional results on code generation, a different backbone, and out-of-domain evaluations further demonstrate the generalization ability of DelTA.

Related papers

• Not All Tokens Deserve Equal Credit: Counterfactual Sensitivity Credit Reallocation for Long-CoT Reasoning [3.9783774144289623]
  Critic-free methods such as GRPO convert response-level rewards into advantages and uniformly broadcast them across tokens.<n>We propose Counterfactual Sensitivity Credit Reallocation (CSCR), a simple extension of GRPO that reduces credit for highly sensitive tokens.<n>On long-CoT mathematical reasoning benchmarks, CSCR consistently outperforms GRPO baseline with the same number of policy updates.
  arXiv  Detail & Related papers  (2026-07-30T09:03:33Z)
• Right in the Right Way: LM Training with Verifiable Rewards and Human Demonstrations [46.90253276172931]
  We propose an adversarial generator-discriminator framework that augments verifiable rewards with a learned signal from human demonstrations.<n>Our approach consistently improves non-verifiable properties while preserving the accuracy gains of RLVR.<n>In bug fixing, our method produces solutions with significantly lower edit distance compared to RLVR baselines.
  arXiv  Detail & Related papers  (2026-07-01T17:13:35Z)
• Which Tokens Matter? Adaptive Token Selection for RLVR with the Relative Surprisal Index [6.642733744927882]
  We argue that evaluating sampled-token probability or entropy in isolation is insufficient to capture the policy optimization dynamics.<n>We introduce the Relative Surprisal Index (RSI), a principled, information-theoretic metric that naturally couples the token's entropy with the probability of the selected token.<n>We show that RSI is related to the local ratio between the first-order variations of the logit-gradient norm and predictive entropy under a selected-logit perturbation.
  arXiv  Detail & Related papers  (2026-06-30T12:33:19Z)
• PRPO: Perception-Reinforced Policy Optimization via Token-Level Dynamic Advantage Reshaping [5.89473045822308]
  Reinforcement Learning with Verifiable Rewards (RLVR) has become an effective paradigm for improving the reasoning capability of Large Vision-Language Models (LVLMs)<n>Existing RLVR methods rely on trajectory-level outcome rewards, which assign identical learning signals across all generated tokens.<n>We propose Perception-Reinforced Policy Optimization (PRPO), a token-level reinforcement learning framework that explicitly identifies and reinforces pivotal perceptual tokens.
  arXiv  Detail & Related papers  (2026-06-07T16:06:29Z)
• Leveraging Error Diversity in Group Rollouts for Reinforcement Learning [61.10980882949414]
  Reinforcement Learning from Verifiable Rewards (RLVR) typically samples multiple responses per prompt and assigns binary rewards based on individual correctness.<n>We propose Error Diversity Advantage Shaping (EDAS) to modulate the advantage signal for incorrect rollouts based on intra-group error diversity.<n>EDAS operates as a simple post-hoc adjustment that can be seamlessly integrated into any RLVR algorithm.
  arXiv  Detail & Related papers  (2026-05-17T08:52:31Z)
• SAGE: Shaping Anchors for Guided Exploration in RLVR of LLMs [55.46289074417954]
  reinforcement learning with verifiable rewards (RLVR) reliably improves pass@1 on reasoning tasks, yet often fails to yield comparable gains in pass@k.<n>We argue that a central structural constraint arises from reverse-KL regularization, which stabilizes training but inherently anchors the policy to the reference distribution.<n>We propose SAGE, a principled framework that enables controllable empirical support expansion by reshaping the reverse-KL anchor distribution itself.
  arXiv  Detail & Related papers  (2026-05-15T07:42:21Z)
• Revisiting Reinforcement Learning with Verifiable Rewards from a Contrastive Perspective [10.958642517467721]
  We present a framework for Contrastive Sequence-level Policy Optimization in RLVR.<n>ConSPO replaces GRPO's clipped ratio-based scores with length-normalized sequence log-probabilities.<n>ConSPO consistently outperforms several strong RLVR baselines on challenging mathematical reasoning benchmarks.
  arXiv  Detail & Related papers  (2026-05-13T04:02:36Z)
• Verifier-Free RL for LLMs via Intrinsic Gradient-Norm Reward [69.99652051809737]
  In this work, we propose Verifier-free Intrinsic Gradient-Norm Reward (VIGOR)<n>VIGOR is a simple reward that uses only the policy model itself.<n>It exhibits cross-domain transfer to code benchmarks when trained only on math data.
  arXiv  Detail & Related papers  (2026-05-11T03:15:37Z)
• The Cancellation Hypothesis in Critic-Free RL: From Outcome Rewards to Token Credits [56.379030343338776]
  We study criticfree RL from a token-level perspective, revealing the token-flipping phenomenon.<n>We show that a token's change in probability is not fully determined by its own advantage; coupled gradient interactions with other tokens also play a non-negligible role.
  arXiv  Detail & Related papers  (2026-05-09T04:07:20Z)
• ResRL: Boosting LLM Reasoning via Negative Sample Projection Residual Reinforcement Learning [60.411097231348975]
  Reinforcement Learning with Verifiable Rewards (RLVR) enhances reasoning of Large Language Models (LLMs)<n>This paper proposes negative sample projection Residual Reinforcement Learning (ResRL) that decouples similar semantic distributions among positive and negative responses.
  arXiv  Detail & Related papers  (2026-05-01T03:57:44Z)
• Overconfident Errors Need Stronger Correction: Asymmetric Confidence Penalties for Reinforcement Learning [17.384089089363382]
  We identify a root cause that existing methods overlook: the uniform penalization of errors.<n>Current approaches treat all incorrect rollouts within a group identically.<n>We propose the Asymmetric Confidence-aware Error Penalty (ACE)
  arXiv  Detail & Related papers  (2026-02-24T22:46:43Z)
• Writing-Zero: Bridge the Gap Between Non-verifiable Tasks and Verifiable Rewards [11.149294285483782]
  We propose a unified RLVR-based training paradigm that bridges the gap between non-verifiable tasks and verifiable rewards.<n>We introduce a writing-principle-based pairwise Generative Reward Model (GenRM) and a novel Bootstrapped Relative Policy Optimization (BRPO) algorithm.<n>Our approach empowers LLMs to develop robust writing capabilities without supervised fine-tuning.
  arXiv  Detail & Related papers  (2025-05-30T14:34:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.