Entropy Regularization Improves Policy Robustness in Continuous-Time Reinforcement Learning

• URL: http://arxiv.org/abs/2607.03168v1
• Date: Fri, 03 Jul 2026 10:09:39 GMT
• Title: Entropy Regularization Improves Policy Robustness in Continuous-Time Reinforcement Learning
• Abstract summary: This paper establishes the first robustness guarantees for entropy-regularized continuous-time Markov decision processes.<n>We analytically characterize the induced robust sets and prove that they expand monotonically with the regularization strength.<n> Experiments on queueing network control and market making confirm our theory.
• Score: 42.77878041827321
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Entropy regularization is widely used in continuous-time reinforcement learning (RL) to reduce sensitivity to environmental perturbations, yet its robustness benefits lack a rigorous theoretical foundation. This paper establishes the first robustness guarantees for entropy-regularized continuous-time Markov decision processes. We show that maximizing an entropy-regularized objective yields a lower bound on a worst-case robust RL problem with joint reward and transition perturbations. We analytically characterize the induced robust sets and prove that they expand monotonically with the regularization strength, justifying the empirical observation that stronger entropy improves robustness. In contrast to prior discrete-time analyses, our results remove the intractable state-distribution entropy term and provide guarantees invariant to action frequency. Experiments on queueing network control and market making confirm our theory, showing that entropy-regularized policies outperform greedy and $ε$-greedy baselines under dynamics perturbations.

Related papers

• Entropy Polarity in Reinforcement Fine-Tuning: Direction, Asymmetry, and Control [77.8471519867791]
  Empirically, we show that entropy polarity reliably predicts entropy changes.<n>We propose Polarity-Aware Policy Optimization (PAPO), which preserves both polarity branches and implements entropy control through advantage reweighting.
  arXiv  Detail & Related papers  (2026-05-12T08:47:05Z)
• Understanding and Preventing Entropy Collapse in RLVR with On-Policy Entropy Flow Optimization [53.75029276020459]
  We propose an adaptive entropy flow balancing mechanism that rescales entropy-increasing and entropy-decreasing updates according to their contributions to entropy change.<n>Experiments on six mathematical reasoning benchmarks demonstrate that OPEFO improves training and final performance.
  arXiv  Detail & Related papers  (2026-05-12T04:08:17Z)
• Rethinking Exploration in RLVR: From Entropy Regularization to Refinement via Bidirectional Entropy Modulation [33.15281300187744]
  Reinforcement learning with rewards has advanced the reasoning capabilities of large language models.<n>It faces a limitation termed textitrestricted exploration, where the policy rapidly converges to a narrow set of solutions.<n>We propose textbfAsymGRPO, an exploratory framework that explicitly decouples the modulation of positive and negative rollouts.
  arXiv  Detail & Related papers  (2026-04-06T17:42:38Z)
• A Comparative Theoretical Analysis of Entropy Control Methods in Reinforcement Learning [3.002731187735679]
  Reinforcement learning (RL) has become a key approach for enhancing reasoning in large language models (LLMs)<n>This paper provides a comparative theoretical analysis of two entropy control strategies: traditional entropy regularization and the recently proposed covariance-based mechanism.
  arXiv  Detail & Related papers  (2026-04-02T18:52:59Z)
• Mind Your Entropy: From Maximum Entropy to Trajectory Entropy-Constrained RL [56.085103402298905]
  We propose a trajectory entropy-constrained reinforcement learning (TECRL) framework to address these two challenges.<n>Within this framework, we first separately learn two Q-functions, one associated with reward and the other with entropy, ensuring clean and stable value targets unaffected by temperature updates.<n>We develop a practical off-policy algorithm, DSAC-E, by extending the state-of-the-art distributional soft actor-critic with three refinements.
  arXiv  Detail & Related papers  (2025-10-25T09:17:47Z)
• Rediscovering Entropy Regularization: Adaptive Coefficient Unlocks Its Potential for LLM Reinforcement Learning [55.59724323303857]
  We propose a framework that balances exploration and exploitation via three components: difficulty-aware coefficient allocation, initial-anchored target entropy, and dynamic global coefficient adjustment.<n>Experiments on multiple mathematical reasoning benchmarks show that AER consistently outperforms baselines, improving both reasoning accuracy and exploration capability.
  arXiv  Detail & Related papers  (2025-10-13T03:10:26Z)
• Rethinking Entropy Interventions in RLVR: An Entropy Change Perspective [11.65148836911294]
  entropy collapse is a rapid loss of policy diversity, stemming from the exploration-exploitation imbalance and leading to a lack of generalization.<n>Recent entropy-intervention methods aim to prevent coloredtextentropy collapse, yet their underlying mechanisms remain unclear.<n>We introduce an entropy-change-aware reweighting scheme, namely Stabilizing Token-level Entropy-changE via Reweighting (STEER)
  arXiv  Detail & Related papers  (2025-10-11T10:17:38Z)
• State Entropy Regularization for Robust Reinforcement Learning [49.08983925413188]
  We show that state entropy regularization improves robustness to structured and spatially correlated perturbations.<n>These types of variation are common in transfer learning but often overlooked by standard robust reinforcement learning methods.
  arXiv  Detail & Related papers  (2025-06-08T11:15:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.