The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models

• URL: http://arxiv.org/abs/2505.22617v1
• Date: Wed, 28 May 2025 17:38:45 GMT
• Title: The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models
• Abstract summary: This paper aims to overcome a major obstacle in scaling RL for reasoning with LLMs, namely the collapse of policy entropy.<n>In practice, we establish a transformation equation R=-a*eH+b between entropy H and downstream performance R.<n>We propose two simple yet effective techniques, namely Clip-Cov and KL-Cov, which clip and apply KL penalty to tokens with high covariances respectively.
• Score: 99.98293908799731
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: This paper aims to overcome a major obstacle in scaling RL for reasoning with LLMs, namely the collapse of policy entropy. Such phenomenon is consistently observed across vast RL runs without entropy intervention, where the policy entropy dropped sharply at the early training stage, this diminished exploratory ability is always accompanied with the saturation of policy performance. In practice, we establish a transformation equation R=-a*e^H+b between entropy H and downstream performance R. This empirical law strongly indicates that, the policy performance is traded from policy entropy, thus bottlenecked by its exhaustion, and the ceiling is fully predictable H=0, R=-a+b. Our finding necessitates entropy management for continuous exploration toward scaling compute for RL. To this end, we investigate entropy dynamics both theoretically and empirically. Our derivation highlights that, the change in policy entropy is driven by the covariance between action probability and the change in logits, which is proportional to its advantage when using Policy Gradient-like algorithms. Empirical study shows that, the values of covariance term and entropy differences matched exactly, supporting the theoretical conclusion. Moreover, the covariance term stays mostly positive throughout training, further explaining why policy entropy would decrease monotonically. Through understanding the mechanism behind entropy dynamics, we motivate to control entropy by restricting the update of high-covariance tokens. Specifically, we propose two simple yet effective techniques, namely Clip-Cov and KL-Cov, which clip and apply KL penalty to tokens with high covariances respectively. Experiments show that these methods encourage exploration, thus helping policy escape entropy collapse and achieve better downstream performance.

Related papers

• Entropy Polarity in Reinforcement Fine-Tuning: Direction, Asymmetry, and Control [77.8471519867791]
  Empirically, we show that entropy polarity reliably predicts entropy changes.<n>We propose Polarity-Aware Policy Optimization (PAPO), which preserves both polarity branches and implements entropy control through advantage reweighting.
  arXiv  Detail & Related papers  (2026-05-12T08:47:05Z)
• Understanding and Preventing Entropy Collapse in RLVR with On-Policy Entropy Flow Optimization [53.75029276020459]
  We propose an adaptive entropy flow balancing mechanism that rescales entropy-increasing and entropy-decreasing updates according to their contributions to entropy change.<n>Experiments on six mathematical reasoning benchmarks demonstrate that OPEFO improves training and final performance.
  arXiv  Detail & Related papers  (2026-05-12T04:08:17Z)
• Rethinking Exploration in RLVR: From Entropy Regularization to Refinement via Bidirectional Entropy Modulation [33.15281300187744]
  Reinforcement learning with rewards has advanced the reasoning capabilities of large language models.<n>It faces a limitation termed textitrestricted exploration, where the policy rapidly converges to a narrow set of solutions.<n>We propose textbfAsymGRPO, an exploratory framework that explicitly decouples the modulation of positive and negative rollouts.
  arXiv  Detail & Related papers  (2026-04-06T17:42:38Z)
• Entropy-Preserving Reinforcement Learning [42.27609126193588]
  We show that many policy gradient algorithms naturally reduce the entropy as part of training.<n>In this paper, we argue that entropy should be actively monitored and controlled throughout training.
  arXiv  Detail & Related papers  (2026-03-12T08:47:40Z)
• Flexible Entropy Control in RLVR with Gradient-Preserving Perspective [19.86794452199207]
  Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a critical method for enhancing the reasoning capabilities of Large Language Models (LLMs)<n>This paper proposes reshaping entropy control in RL from the perspective of Gradient-Preserving Clipping.<n>We introduce a novel regulation mechanism using dynamic clipping threshold to precisely manage entropy.
  arXiv  Detail & Related papers  (2026-02-10T13:42:12Z)
• On the Entropy Dynamics in Reinforcement Fine-Tuning of Large Language Models [54.61810451777578]
  Entropy serves as a critical metric for measuring the diversity of outputs generated by large language models.<n>Recent studies increasingly focus on monitoring and adjusting entropy to better balance exploration and exploitation in reinforcement fine-tuning.
  arXiv  Detail & Related papers  (2026-02-03T11:14:58Z)
• Entropy Ratio Clipping as a Soft Global Constraint for Stable Reinforcement Learning [49.92803982100042]
  We propose using the entropy ratio between the current and previous policies as a new global metric.<n>We introduce an textbfEntropy Ratio (ERC) mechanism that imposes bidirectional constraints on the entropy ratio.<n>This stabilizes policy updates at the global distribution level and compensates for the inability of PPO-clip to regulate probability shifts of un-sampled actions.
  arXiv  Detail & Related papers  (2025-12-05T10:26:32Z)
• Revisiting Entropy in Reinforcement Learning for Large Reasoning Models [54.96908589622163]
  We investigate the entropy dynamics of large language models trained withReinforcement learning with verifiable rewards (RLVR)<n>Our findings reveal that the number of off-policy updates, the diversity of training data, and the clipping thresholds in the optimization objective are critical factors influencing the entropy of LLMs trained with RLVR.
  arXiv  Detail & Related papers  (2025-11-08T12:50:41Z)
• Mind Your Entropy: From Maximum Entropy to Trajectory Entropy-Constrained RL [56.085103402298905]
  We propose a trajectory entropy-constrained reinforcement learning (TECRL) framework to address these two challenges.<n>Within this framework, we first separately learn two Q-functions, one associated with reward and the other with entropy, ensuring clean and stable value targets unaffected by temperature updates.<n>We develop a practical off-policy algorithm, DSAC-E, by extending the state-of-the-art distributional soft actor-critic with three refinements.
  arXiv  Detail & Related papers  (2025-10-25T09:17:47Z)
• Rediscovering Entropy Regularization: Adaptive Coefficient Unlocks Its Potential for LLM Reinforcement Learning [55.59724323303857]
  We propose a framework that balances exploration and exploitation via three components: difficulty-aware coefficient allocation, initial-anchored target entropy, and dynamic global coefficient adjustment.<n>Experiments on multiple mathematical reasoning benchmarks show that AER consistently outperforms baselines, improving both reasoning accuracy and exploration capability.
  arXiv  Detail & Related papers  (2025-10-13T03:10:26Z)
• Rethinking Entropy Interventions in RLVR: An Entropy Change Perspective [11.65148836911294]
  entropy collapse is a rapid loss of policy diversity, stemming from the exploration-exploitation imbalance and leading to a lack of generalization.<n>Recent entropy-intervention methods aim to prevent coloredtextentropy collapse, yet their underlying mechanisms remain unclear.<n>We introduce an entropy-change-aware reweighting scheme, namely Stabilizing Token-level Entropy-changE via Reweighting (STEER)
  arXiv  Detail & Related papers  (2025-10-11T10:17:38Z)
• Arbitrary Entropy Policy Optimization: Entropy Is Controllable in Reinforcement Fine-tuning [36.00460460149206]
  We propose Arbitrary Entropy Policy Optimization (AEPO), which eliminates entropy collapse by replacing entropy bonuses with REINFORCE policy gradient on temperature-adjusted distributions.<n>AEPO integrates three key designs: policy gradient as regularization, distribution as regularization, and REINFORCE as regularization, enabling precise entropy control without distorting optimization.
  arXiv  Detail & Related papers  (2025-10-09T12:24:08Z)
• Clip-Low Increases Entropy and Clip-High Decreases Entropy in Reinforcement Learning of Large Language Models [29.822717720666134]
  We show that the clipping mechanism in PPO and GRPO induces biases on entropy.<n>With a more aggressive clip-low value, one can increase entropy, promote exploration, and ultimately prevent entropy collapse in RLVR training.
  arXiv  Detail & Related papers  (2025-09-30T11:33:15Z)
• Complexity-Driven Policy Optimization [5.757876638037014]
  We propose replacing the entropy bonus with a more robust complexity bonus.<n>This regularizer encourages policies that balance quantity (high entropy) with structure (high disequilibrium)<n>Such behaviors arise because the regularizer suppresses both extremes, e.g., maximal disorder and complete order, creating pressure for agents to discover structured yet adaptable strategies.
  arXiv  Detail & Related papers  (2025-09-24T19:32:03Z)
• Decomposing the Entropy-Performance Exchange: The Missing Keys to Unlocking Effective Reinforcement Learning [106.68304931854038]
  Reinforcement learning with verifiable rewards (RLVR) has been widely used for enhancing the reasoning abilities of large language models (LLMs)<n>We conduct a systematic empirical analysis of the entropy-performance exchange mechanism of RLVR across different levels of granularity.<n>Our analysis reveals that, in the rising stage, entropy reduction in negative samples facilitates the learning of effective reasoning patterns.<n>In the plateau stage, learning efficiency strongly correlates with high-entropy tokens present in low-perplexity samples and those located at the end of sequences.
  arXiv  Detail & Related papers  (2025-08-04T10:08:10Z)
• State Entropy Regularization for Robust Reinforcement Learning [49.08983925413188]
  We show that state entropy regularization improves robustness to structured and spatially correlated perturbations.<n>These types of variation are common in transfer learning but often overlooked by standard robust reinforcement learning methods.
  arXiv  Detail & Related papers  (2025-06-08T11:15:31Z)
• Entropy annealing for policy mirror descent in continuous time and space [2.8255028200738455]
  We study the impact of entropy regularization on the convergence of policy methods for exit time control problems.<n>This paper explains how entropy regularization improves policy optimization, even with the true gradient.
  arXiv  Detail & Related papers  (2024-05-30T17:02:18Z)
• Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimization [55.97310586039358]
  Diffusion models have garnered widespread attention in Reinforcement Learning (RL) for their powerful expressiveness and multimodality.<n>We propose a novel model-free diffusion-based online RL algorithm, Q-weighted Variational Policy Optimization (QVPO)<n>Specifically, we introduce the Q-weighted variational loss, which can be proved to be a tight lower bound of the policy objective in online RL under certain conditions.<n>We also develop an efficient behavior policy to enhance sample efficiency by reducing the variance of the diffusion policy during online interactions.
  arXiv  Detail & Related papers  (2024-05-25T10:45:46Z)
• Diffusion Actor-Critic with Entropy Regulator [32.79341490514616]
  We propose an online RL algorithm termed diffusion actor-critic with entropy regulator (DACER)<n>This algorithm conceptualizes the reverse process of the diffusion model as a novel policy function.<n>Experiments on MuJoCo benchmarks and a multimodal task demonstrate that the DACER algorithm achieves state-of-the-art (SOTA) performance.
  arXiv  Detail & Related papers  (2024-05-24T03:23:27Z)
• On the Global Convergence of Policy Gradient in Average Reward Markov Decision Processes [50.68789924454235]
  We present the first finite time global convergence analysis of policy gradient in the context of average reward Markov decision processes (MDPs) Our analysis shows that the policy gradient iterates converge to the optimal policy at a sublinear rate of $Oleft(frac1Tright),$ which translates to $Oleft(log(T)right)$ regret, where $T$ represents the number of iterations.
  arXiv  Detail & Related papers  (2024-03-11T15:25:03Z)
• Predictable Reinforcement Learning Dynamics through Entropy Rate Minimization [16.335645061396455]
  In Reinforcement Learning (RL), agents have no incentive to exhibit predictable behaviors.<n>We propose a novel method to induce predictable behavior in RL agents, termed Predictability-Aware RL (PARL)<n>Our method maximizes a linear combination of a standard discounted reward and the negative entropy rate, thus trading off optimality with predictability.
  arXiv  Detail & Related papers  (2023-11-30T16:53:32Z)
• Robust Policy Optimization in Deep Reinforcement Learning [16.999444076456268]
  In continuous action domains, parameterized distribution of action distribution allows easy control of exploration. In particular, we propose an algorithm called Robust Policy Optimization (RPO), which leverages a perturbed distribution. We evaluated our methods on various continuous control tasks from DeepMind Control, OpenAI Gym, Pybullet, and IsaacGym.
  arXiv  Detail & Related papers  (2022-12-14T22:43:56Z)
• Optimal scheduling of entropy regulariser for continuous-time linear-quadratic reinforcement learning [9.779769486156631]
  Herein agent interacts with the environment by generating noisy controls distributed according to the optimal relaxed policy. This exploration-exploitation trade-off is determined by the strength of entropy regularisation. We prove that the regret, for both learning algorithms, is of the order $mathcalO(sqrtN) $ (up to a logarithmic factor) over $N$ episodes, matching the best known result from the literature.
  arXiv  Detail & Related papers  (2022-08-08T23:36:40Z)
• Action Redundancy in Reinforcement Learning [54.291331971813364]
  We show that transition entropy can be described by two terms; namely, model-dependent transition entropy and action redundancy. Our results suggest that action redundancy is a fundamental problem in reinforcement learning.
  arXiv  Detail & Related papers  (2021-02-22T19:47:26Z)
• Entropy-Augmented Entropy-Regularized Reinforcement Learning and a Continuous Path from Policy Gradient to Q-Learning [5.185562073975834]
  entropy augmentation is reformulated and leads to a motivation to introduce an additional entropy term to the objective function. It results in a policy which monotonically improves while interpolating from the current policy to the softmax greedy policy.
  arXiv  Detail & Related papers  (2020-05-18T16:15:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.