Not All Tokens Learn Alike: Attention Entropy Reveals Heterogeneous Signals in RL Reasoning

• URL: http://arxiv.org/abs/2605.07660v1
• Date: Fri, 08 May 2026 12:31:28 GMT
• Title: Not All Tokens Learn Alike: Attention Entropy Reveals Heterogeneous Signals in RL Reasoning
• Abstract summary: Reinforcement-learning-based post-training has become a key approach for improving the reasoning ability of large language models.<n>This work studies their heterogeneity through attention entropy, which measures how concentrated or diffuse the contextual support is for each response token.
• Score: 11.322734738973603
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement-learning-based post-training has become a key approach for improving the reasoning ability of large language models, but its token-level learning signals remain poorly understood. This work studies their heterogeneity through attention entropy, which measures how concentrated or diffuse the contextual support is for each response token. We first show that token-level RL objectives are sparsely estimable: uniformly random 20 percent token subsets preserve much of the full-token held-out performance, suggesting substantial redundancy in token-level updates. However, entropy-structured subsets behave very differently. Low-attention-entropy tokens, which we call anchors, rely on concentrated support, produce stable gradients aligned with full-token updates, and provide a reliable optimization backbone, but tend to plateau on harder benchmarks. High-attention-entropy tokens, which we call explorers, aggregate more diffuse context and induce larger but more volatile gradients. Explorer-only training is unstable on average, though rare successful runs suggest that these tokens may contain useful hard-reasoning signals when optimization remains stable. We support this anchor-explorer spectrum with evidence-gathering analyses, entropy dynamics, gradient-geometry diagnostics, and controls showing that position, predictive entropy, and loss normalization do not explain the observed asymmetry. Finally, a dynamic entropy-aware soft-reweighting intervention improves Qwen3-8B-Base from 34.39 to 37.40 held-out average in the strongest setting. These findings suggest that attention entropy reveals optimization-relevant structure in token-level RL signals, and that uniform token averaging can obscure meaningful heterogeneity in reasoning post-training.

Related papers

• Which Tokens Matter? Adaptive Token Selection for RLVR with the Relative Surprisal Index [6.642733744927882]
  We argue that evaluating sampled-token probability or entropy in isolation is insufficient to capture the policy optimization dynamics.<n>We introduce the Relative Surprisal Index (RSI), a principled, information-theoretic metric that naturally couples the token's entropy with the probability of the selected token.<n>We show that RSI is related to the local ratio between the first-order variations of the logit-gradient norm and predictive entropy under a selected-logit perturbation.
  arXiv  Detail & Related papers  (2026-06-30T12:33:19Z)
• STARE: Surprisal-Guided Token-Level Advantage Reweighting for Policy Entropy Stability [78.63876433996107]
  We conduct a first-order gradient analysis of token-level entropy dynamics under GRPO.<n>We propose STARE, which identifies entropy-critical token subsets via batch-internal surprisal quantiles.<n>On AIME24 and AIME25, STARE outperforms DAPO and other competitive baselines by 4%-8% in average accuracy.
  arXiv  Detail & Related papers  (2026-06-17T16:13:42Z)
• Entropy Is Not Enough: Unlocking Effective Reinforcement Learning for Visual Reasoning via Vision-Anchored Token Selection [68.2118210672375]
  We introduce VEPO (Vision-Entropy token-selection for Policy Optimization), an effective RL framework explicitly integrating visual sensitivity with token entropy.<n>Our experiments demonstrate VEPO's leading performance, significantly outperforming the entropy-only baseline by 2.28 points at 7B-scale and 3.15 points at 3B-scale.
  arXiv  Detail & Related papers  (2026-06-02T17:26:55Z)
• Taming the Thinker: Conditional Entropy Shaping for Adaptive LLM Reasoning [63.36243250685778]
  Conditional Entropy Shaping (CES) is a framework that dynamically controls token-level response entropy.<n>CES consistently improves average accuracy while reducing response length relative to DAPO.
  arXiv  Detail & Related papers  (2026-05-19T04:41:51Z)
• Rethinking Token-Level Credit Assignment in RLVR: A Polarity-Entropy Analysis [33.07421874137999]
  Reinforcement Learning with Verifiable Rewards (RLVR) has substantially improved the reasoning ability of Large Language Models (LLMs)<n>We analyze this problem through the joint lens of reward polarity and token entropy.<n>We propose Entropy-Aware Policy Optimization (EAPO) that modulates token-level learning signals accordingly.
  arXiv  Detail & Related papers  (2026-04-13T06:32:49Z)
• Early Decisions Matter: Proximity Bias and Initial Trajectory Shaping in Non-Autoregressive Diffusion Language Models [54.16797570104461]
  Diffusion-based language models (dLLMs) have emerged as a promising alternative to autoregressive language models.<n>We investigate non-autoregressive decoding in dLLMs by systematically analyzing its inference dynamics along the temporal axis.
  arXiv  Detail & Related papers  (2026-04-12T10:26:41Z)
• STAPO: Stabilizing Reinforcement Learning for LLMs by Silencing Rare Spurious Tokens [38.425692691443764]
  ExistingReinforcement Learning (RL) fine-tuning methods rely heavily on entropy regularization and reweighting to maintain stability.<n>In practice, they often suffer from late-stage performance collapse, leading to degraded reasoning quality and unstable training.<n>We find that training instability can be caused by a tiny fraction of tokens, approximately 0.01%, which we term spurious tokens.<n>We propose Spurious-Token-Aware Policy Optimization (STAPO), which promotes stable and effective large-scale model refinement.
  arXiv  Detail & Related papers  (2026-02-17T14:46:48Z)
• Probability-Entropy Calibration: An Elastic Indicator for Adaptive Fine-tuning [55.2818264614932]
  RankTuner introduces a probability--entropy calibration signal, the Relative Rank Indicator, which compares the rank of the ground-truth token with its expected rank under the prediction distribution.<n>The inverse indicator is used as a token-wise Relative Scale to reweight the fine-tuning objective, focusing updates on truly under-learned tokens.
  arXiv  Detail & Related papers  (2026-02-02T07:27:19Z)
• EDIS: Diagnosing LLM Reasoning via Entropy Dynamics [3.858418431840288]
  We show that the emphtemporal evolution of confidence during generation carries richer information than aggregate statistics alone.<n>We introduce the Entropy Dynamics Instability Score (textbfEDIS), a trajectory-level metric quantifying instability in entropy evolution.
  arXiv  Detail & Related papers  (2026-02-01T15:43:50Z)
• Attention Illuminates LLM Reasoning: The Preplan-and-Anchor Rhythm Enables Fine-Grained Policy Optimization [56.083511902353365]
  Reinforcement learning (RL) typically applies uniform credit across an entire generation of Large language models.<n>This work positions attention as a privileged substrate that renders the internal logic of LLMs as a mechanistic blueprint of reasoning itself.<n>We introduce three novel RL strategies that dynamically perform targeted credit assignment to critical nodes.
  arXiv  Detail & Related papers  (2025-10-15T13:49:51Z)
• From Uniform to Heterogeneous: Tailoring Policy Optimization to Every Token's Nature [38.46122853450324]
  Existing algorithms apply uniform optimization to all tokens, ignoring their different roles in reasoning process.<n>We introduce Heterogeneous Adaptive Policy Optimization (HAPO), a token-aware algorithm that dynamically adapts optimization based on token entropy.
  arXiv  Detail & Related papers  (2025-09-20T09:30:25Z)
• Emergent Hierarchical Reasoning in LLMs through Reinforcement Learning [56.496001894673235]
  Reinforcement Learning (RL) has proven highly effective at enhancing the complex reasoning abilities of Large Language Models (LLMs)<n>Our analysis reveals that puzzling phenomena like aha moments", length-scaling'' and entropy dynamics are not disparate occurrences but hallmarks of an emergent reasoning hierarchy.
  arXiv  Detail & Related papers  (2025-09-03T18:52:49Z)
• Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning [80.87085014818052]
  Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a powerful approach to enhancing the reasoning capabilities of Large Language Models (LLMs)<n>In this work, we undertake a pioneering exploration of RLVR through the novel perspective of token entropy patterns.<n>We observe that only a small fraction of tokens exhibit high entropy, and these tokens act as critical forks that steer the model toward diverse reasoning pathways.
  arXiv  Detail & Related papers  (2025-06-02T17:54:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.