STARE: Surprisal-Guided Token-Level Advantage Reweighting for Policy Entropy Stability

• URL: http://arxiv.org/abs/2606.19236v1
• Date: Wed, 17 Jun 2026 16:13:42 GMT
• Title: STARE: Surprisal-Guided Token-Level Advantage Reweighting for Policy Entropy Stability
• Abstract summary: We conduct a first-order gradient analysis of token-level entropy dynamics under GRPO.<n>We propose STARE, which identifies entropy-critical token subsets via batch-internal surprisal quantiles.<n>On AIME24 and AIME25, STARE outperforms DAPO and other competitive baselines by 4%-8% in average accuracy.
• Score: 78.63876433996107
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement Learning with Verifiable Rewards algorithms like GRPO have emerged as the dominant post-training paradigm for complex reasoning in LLMs, yet commonly suffer from policy entropy collapse during training. We conduct a first-order gradient analysis of token-level entropy dynamics under GRPO and identify a token-level credit assignment mismatch: the per-token entropy variation decomposes into the product of the trajectory-level advantage and an entropy sensitivity function over the next-token distribution, yielding an advantage-surprisal four-quadrant structure and a near-criticality property. Motivated by it, we propose STARE (Surprisal-guided Token-level Advantage Reweighting for policy Entropy stability), which identifies entropy-critical token subsets via batch-internal surprisal quantiles, selectively reweights their effective advantages, and incorporates a target-entropy closed-loop gate for stable entropy regulation. Across model scales from 1.5B to 32B and three task families (Short CoT, Long CoT, and Multi-Turn Tool Use), STARE sustains stable RL training over thousands of steps while maintaining policy entropy within the target band. On AIME24 and AIME25, STARE outperforms DAPO and other competitive baselines by 4%-8% in average accuracy, with reflection tokens and response length growing in tandem, indicating sustained exploration-exploitation balance that further unlocks RL training potential.Code is available at https://github.com/hp-luo/STARE.

Related papers

• ACPO: Adaptive Credit Policy Optimization via Fine-Grained Surrogate Entropy [45.28312869386833]
  We propose a token-level credit assignment framework based on a mode-local surrogate entropy.<n>ACPO asymmetrically modulates policy updates by emphasizing uncertain decisions in successful rollouts and overconfident tokens in failed rollouts.<n>Experiments on mathematical reasoning and coding benchmarks, including AIME 2025 and HumanEvalPro, show that ACPO consistently improves over strong RL baselines.
  arXiv  Detail & Related papers  (2026-07-03T09:14:27Z)
• Beyond Entropy: Learning from Token-Level Distributional Deviations for LLM Reasoning [36.34371525731833]
  We introduce the Independent Combinatorial Tokens (ICT) framework, which shifts the optimization focus from scalar uncertainty to the distributional properties of token logits.<n>Our theoretical analysis, grounded in both Shannon and second-order Rényi entropy, proves that selectively updating on these tokens regulates policy concentration.<n> Empirical results demonstrate that updating only the top 10% of unique tokens on Qwen2.5 (0.5B/1.5B/7B) models yields an average pass@4 improvement of 4.58%, with a maximum gain of 14.9%, over GRPO, 20-Entropy, and STAPO baselines.
  arXiv  Detail & Related papers  (2026-06-18T04:11:56Z)
• OPPO: Bayesian Value Recursion for Token-Level Credit Assignment in LLM Reasoning [17.98540130851038]
  Reinforcement learning with verifiable rewards has become the standard recipe for improving LLM reasoning.<n>But the dominant algorithm GRPO assigns a single trajectory-level advantage to every token.<n>We propose Oracle-Prompted Policy Optimization (OPPO), which rests on a single observation.
  arXiv  Detail & Related papers  (2026-05-21T00:55:13Z)
• Entropy Polarity in Reinforcement Fine-Tuning: Direction, Asymmetry, and Control [77.8471519867791]
  Empirically, we show that entropy polarity reliably predicts entropy changes.<n>We propose Polarity-Aware Policy Optimization (PAPO), which preserves both polarity branches and implements entropy control through advantage reweighting.
  arXiv  Detail & Related papers  (2026-05-12T08:47:05Z)
• Token-Level Policy Optimization: Linking Group-Level Rewards to Token-Level Aggregation via Sequence-Level Likelihood [9.53011880338711]
  We propose TEPO, a novel token-level framework that leverages sequence-level likelihood to link group-level rewards with individual tokens.<n>Experiments demonstrate that TEPO achieves state-of-the-art performance on mathematical reasoning benchmarks.<n>It also markedly enhances training stability, reducing convergence time by 50% compared with GRPO/DAPO.
  arXiv  Detail & Related papers  (2026-04-14T13:50:31Z)
• Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing [79.88256756334327]
  Self-Distillation Policy Optimization (SDPO) addresses this by providing denser, more targeted logit-level supervision.<n>We propose Sample-Routed Policy Optimization (SRPO)<n>SRPO routes correct samples to GRPO's reward-aligned reinforcement and failed samples to SDPO's targeted logit-level correction.
  arXiv  Detail & Related papers  (2026-04-02T17:29:18Z)
• STAPO: Stabilizing Reinforcement Learning for LLMs by Silencing Rare Spurious Tokens [38.425692691443764]
  ExistingReinforcement Learning (RL) fine-tuning methods rely heavily on entropy regularization and reweighting to maintain stability.<n>In practice, they often suffer from late-stage performance collapse, leading to degraded reasoning quality and unstable training.<n>We find that training instability can be caused by a tiny fraction of tokens, approximately 0.01%, which we term spurious tokens.<n>We propose Spurious-Token-Aware Policy Optimization (STAPO), which promotes stable and effective large-scale model refinement.
  arXiv  Detail & Related papers  (2026-02-17T14:46:48Z)
• Rethinking the Trust Region in LLM Reinforcement Learning [72.25890308541334]
  Proximal Policy Optimization (PPO) serves as the de facto standard algorithm for Large Language Models (LLMs)<n>We propose Divergence Proximal Policy Optimization (DPPO), which substitutes clipping with a more principled constraint.<n>DPPO achieves superior training and efficiency compared to existing methods, offering a more robust foundation for RL-based fine-tuning.
  arXiv  Detail & Related papers  (2026-02-04T18:59:04Z)
• Agentic Entropy-Balanced Policy Optimization [114.90524574220764]
  Agentic Reinforcement Learning (Agentic RL) has made significant progress in incentivizing the multi-turn, long-horizon tool-use capabilities of web agents.<n>While RL algorithms autonomously explore high-uncertainty tool-call steps under the guidance of entropy, excessive reliance on entropy signals can impose further constraints.<n>We propose the Agentic Entropy-Balanced Policy Optimization (AEPO), an agentic RL algorithm designed to balance entropy in both the rollout and policy update phases.
  arXiv  Detail & Related papers  (2025-10-16T10:40:52Z)
• Rediscovering Entropy Regularization: Adaptive Coefficient Unlocks Its Potential for LLM Reinforcement Learning [55.59724323303857]
  We propose a framework that balances exploration and exploitation via three components: difficulty-aware coefficient allocation, initial-anchored target entropy, and dynamic global coefficient adjustment.<n>Experiments on multiple mathematical reasoning benchmarks show that AER consistently outperforms baselines, improving both reasoning accuracy and exploration capability.
  arXiv  Detail & Related papers  (2025-10-13T03:10:26Z)
• CE-GPPO: Coordinating Entropy via Gradient-Preserving Clipping Policy Optimization in Reinforcement Learning [28.02073546326571]
  Policy entropy reflects the balance between exploration and exploitation during training.<n>Existing methods discard valuable gradient signals from low-probability tokens due to the clipping mechanism.<n>We propose textbfCoordinating textbfEntropy via textbfGradient textbfPreserving textbfPolicy textbfOptimization.
  arXiv  Detail & Related papers  (2025-09-25T03:22:04Z)
• GTPO and GRPO-S: Token and Sequence-Level Reward Shaping with Policy Entropy [5.691990020513277]
  We propose Dynamic Entropy Weighting, a novel mechanism that facilitates fine-grained rewards through two new algorithms.<n>By repurposing policy entropy for reward shaping, we achieve true per-token credit assignment.
  arXiv  Detail & Related papers  (2025-08-06T11:42:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.