Rethinking Token-Level Credit Assignment in RLVR: A Polarity-Entropy Analysis

• URL: http://arxiv.org/abs/2604.11056v1
• Date: Mon, 13 Apr 2026 06:32:49 GMT
• Title: Rethinking Token-Level Credit Assignment in RLVR: A Polarity-Entropy Analysis
• Abstract summary: Reinforcement Learning with Verifiable Rewards (RLVR) has substantially improved the reasoning ability of Large Language Models (LLMs)<n>We analyze this problem through the joint lens of reward polarity and token entropy.<n>We propose Entropy-Aware Policy Optimization (EAPO) that modulates token-level learning signals accordingly.
• Score: 33.07421874137999
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) has substantially improved the reasoning ability of Large Language Models (LLMs). However, its sparse outcome-based rewards pose a fundamental credit assignment problem. We analyze this problem through the joint lens of reward polarity and token entropy. Our diagnostic tool, the Four Quadrant Decomposition, isolates token updates by polarity and entropy, and controlled ablations show that reasoning improvements concentrate in the high-entropy quadrants. To justify this observation theoretically, we adapt Conditional Mutual Information to the autoregressive RLVR setting and prove that the credit a token can carry is upper-bounded by its entropy. This view yields testable predictions that reasoning gains arise primarily from high-entropy tokens, with unique roles for positive and negative updates. A gradient analysis of GRPO further reveals how uniform reward broadcast dilutes signal at high-entropy positions while over-crediting deterministic tokens. Grounded in these insights, we propose Entropy-Aware Policy Optimization (EAPO) that modulates token-level learning signals accordingly. Extensive experiments demonstrate that EAPO outperforms strong baselines across two model families.

Related papers

• ERR+: Sequential Entropy Resolution for Efficient and Decisive LLM Reasoning [13.033342249823022]
  We propose a two-phase RLVR framework for large reasoning models.<n>The first phase trains with the Entropy Relief Reward (ERR), a bonus proportional to cumulative token-level entropy drops in the thinking phase.<n>The second phase introduces the Robust Relative Efficiency Reward, which scores each response's length against co-generated peers.
  arXiv  Detail & Related papers  (2026-08-28T18:28:50Z)
• Which Tokens Matter? Adaptive Token Selection for RLVR with the Relative Surprisal Index [6.642733744927882]
  We argue that evaluating sampled-token probability or entropy in isolation is insufficient to capture the policy optimization dynamics.<n>We introduce the Relative Surprisal Index (RSI), a principled, information-theoretic metric that naturally couples the token's entropy with the probability of the selected token.<n>We show that RSI is related to the local ratio between the first-order variations of the logit-gradient norm and predictive entropy under a selected-logit perturbation.
  arXiv  Detail & Related papers  (2026-06-30T12:33:19Z)
• STARE: Surprisal-Guided Token-Level Advantage Reweighting for Policy Entropy Stability [78.63876433996107]
  We conduct a first-order gradient analysis of token-level entropy dynamics under GRPO.<n>We propose STARE, which identifies entropy-critical token subsets via batch-internal surprisal quantiles.<n>On AIME24 and AIME25, STARE outperforms DAPO and other competitive baselines by 4%-8% in average accuracy.
  arXiv  Detail & Related papers  (2026-06-17T16:13:42Z)
• Taming the Thinker: Conditional Entropy Shaping for Adaptive LLM Reasoning [63.36243250685778]
  Conditional Entropy Shaping (CES) is a framework that dynamically controls token-level response entropy.<n>CES consistently improves average accuracy while reducing response length relative to DAPO.
  arXiv  Detail & Related papers  (2026-05-19T04:41:51Z)
• Entropy Polarity in Reinforcement Fine-Tuning: Direction, Asymmetry, and Control [77.8471519867791]
  Empirically, we show that entropy polarity reliably predicts entropy changes.<n>We propose Polarity-Aware Policy Optimization (PAPO), which preserves both polarity branches and implements entropy control through advantage reweighting.
  arXiv  Detail & Related papers  (2026-05-12T08:47:05Z)
• Not All Tokens Learn Alike: Attention Entropy Reveals Heterogeneous Signals in RL Reasoning [11.322734738973603]
  Reinforcement-learning-based post-training has become a key approach for improving the reasoning ability of large language models.<n>This work studies their heterogeneity through attention entropy, which measures how concentrated or diffuse the contextual support is for each response token.
  arXiv  Detail & Related papers  (2026-05-08T12:31:28Z)
• ERPO: Token-Level Entropy-Regulated Policy Optimization for Large Reasoning Models [3.463914032107119]
  We show that Group Relative Policy Optimization (GRPO) typically assigns a uniform, sequence-level advantage to all tokens.<n>We propose Entropy-Regulated Policy Optimization (ERPO), which transitions the optimization focus from coarse sequences to fine-grained token dynamics.
  arXiv  Detail & Related papers  (2026-03-30T09:20:25Z)
• Beyond High-Entropy Exploration: Correctness-Aware Low-Entropy Segment-Based Advantage Shaping for Reasoning LLMs [6.948242693954442]
  Reinforcement Learning with Verifiable Rewards (RLVR) has become a central approach for improving the reasoning ability of large language models.<n>We propose LESS, a correctness-aware reinforcement framework that performs fine-grained advantage modulation over low-entropy segments.
  arXiv  Detail & Related papers  (2025-11-30T14:19:36Z)
• Revisiting Entropy in Reinforcement Learning for Large Reasoning Models [54.96908589622163]
  We investigate the entropy dynamics of large language models trained withReinforcement learning with verifiable rewards (RLVR)<n>Our findings reveal that the number of off-policy updates, the diversity of training data, and the clipping thresholds in the optimization objective are critical factors influencing the entropy of LLMs trained with RLVR.
  arXiv  Detail & Related papers  (2025-11-08T12:50:41Z)
• Rediscovering Entropy Regularization: Adaptive Coefficient Unlocks Its Potential for LLM Reinforcement Learning [55.59724323303857]
  We propose a framework that balances exploration and exploitation via three components: difficulty-aware coefficient allocation, initial-anchored target entropy, and dynamic global coefficient adjustment.<n>Experiments on multiple mathematical reasoning benchmarks show that AER consistently outperforms baselines, improving both reasoning accuracy and exploration capability.
  arXiv  Detail & Related papers  (2025-10-13T03:10:26Z)
• Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning [80.87085014818052]
  Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a powerful approach to enhancing the reasoning capabilities of Large Language Models (LLMs)<n>In this work, we undertake a pioneering exploration of RLVR through the novel perspective of token entropy patterns.<n>We observe that only a small fraction of tokens exhibit high entropy, and these tokens act as critical forks that steer the model toward diverse reasoning pathways.
  arXiv  Detail & Related papers  (2025-06-02T17:54:39Z)
• Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM's Reasoning Capability [53.51560766150442]
  Critical tokens are elements within reasoning trajectories that significantly influence incorrect outcomes.<n>We present a novel framework for identifying these tokens through rollout sampling.<n>We show that identifying and replacing critical tokens significantly improves model accuracy.
  arXiv  Detail & Related papers  (2024-11-29T18:58:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.