Beyond the Stability-Exploration Dilemma: Environmental Regularization for LLM Policy Optimization

• URL: http://arxiv.org/abs/2608.23311v2
• Date: Tue, 25 Aug 2026 15:31:13 GMT
• Title: Beyond the Stability-Exploration Dilemma: Environmental Regularization for LLM Policy Optimization
• Abstract summary: Policy-KL constrains response behavior and consumes the action-side exploration budget.<n>We argue for an alternative that breaks the dilemma by moving regularization to the input side.<n>ERPO replaces the standard Policy-KL regularizer while achieving effective control over query distribution drift.
• Score: 27.121081058414376
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Policy optimization (PO) for Large Language Models faces a stability--exploration trade-off, currently mediated by an action-side Policy-KL regularizer. This puts practitioners in a double bind: keeping Policy-KL constrains response behavior and consumes the action-side exploration budget, while dropping it leaves the optimization without an explicit drift control. We argue for an alternative that breaks the dilemma by moving regularization to the input side. As training progresses, the distribution over training queries induced by the current policy drifts unchecked from its pre-RL reference distribution. Concretely, Environment-Regularized Policy Optimization (ERPO) introduces a Query-KL (QKL) term that bounds this query distribution shift, together with a dataset-static reference-derived per-query weight that biases each per-query update toward queries typical under the reference. The QKL gradient flows strictly through the query likelihood; the response score function used by policy-gradient estimators does not appear in the QKL term, so QKL exerts no direct gradient pressure on the response distribution---exploration is preserved. ERPO plugs into GRPO/PPO/REINFORCE-style pipelines without additional forward passes. On six mathematical reasoning benchmarks, ERPO replaces the standard Policy-KL regularizer while achieving effective control over query distribution drift, delivering stronger accuracy and substantially more stable behavior under high-temperature decoding and long-horizon training. Our source code are available at https://github.com/AlibabaResearch/ERPO

Related papers

• Toward Plasticity-Preserving KL Regularization for Capability Retention in LLM Reinforcement Learning [43.981833556858135]
  We propose a conditional regularization framework that narrows the preservation constraint from the full output distribution to correctness-conditioned response distributions.<n>CoKL achieves a more favorable balance between target-task improvement and prior-capability retention than existing regularization methods.
  arXiv  Detail & Related papers  (2026-08-03T06:10:33Z)
• Rethinking the Divergence Regularization in LLM RL [56.952725733148746]
  Reinforcement learning (RL) has become a key component of post-training large language models (LLMs)<n>We propose Divergence Regularized Policy Optimization (DRPO), which replaces the hard mask with a smooth advantage-weighted quadratic regularizer on policy shift.<n>DRPO preserves the same trust-region geometry as DPPO while inducing bounded, continuous gradient weights.
  arXiv  Detail & Related papers  (2026-06-08T17:58:23Z)
• One-Way Policy Optimization for Self-Evolving LLMs [63.8638342097375]
  Reinforcement Learning with Verifiable Rewards (RLVR) has become a promising paradigm for scaling reasoning capabilities of Large Language Models (LLMs)<n>We propose One-Way Policy Optimization (OWPO), a method based on the principle of decoupling optimization direction from update magnitude.<n> Experimental results demonstrate that OWPO outperforms strong baselines, including DAPO, OPD, and MOPD.
  arXiv  Detail & Related papers  (2026-05-21T08:25:27Z)
• Stochastic MeanFlow Policies: One-Step Generative Control with Entropic Mirror Descent [53.828537014796574]
  Online off-policy reinforcement learning (RL) is shaped by two choices: the policy class and the update rule.<n>We propose MeanFlow Policies, a one-step generative policy class that maps noise to actions through a MeanFlow transformation.<n>Across seven MuJoCo benchmarks, S improves over Gaussian and generative baselines while retaining single-step inference efficiency.
  arXiv  Detail & Related papers  (2026-05-20T15:14:14Z)
• Flow Matching Policy with Entropy Regularization [16.47598359293598]
  Flow Matching Policy with Entropy Regularization (FMER) is an Ordinary Differential Equation (ODE)-based online RL framework.<n>FMER parameterizes the policy via flow matching and samples actions along a straight probability path, motivated by optimal transport.<n> Experiments on sparse multi-goal FrankaKitchen benchmarks demonstrate that FMER outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-03-18T13:00:20Z)
• Improving Search Agent with One Line of Code [68.58667107354253]
  Tool-based Agentic Reinforcement Learning (TARL) has emerged as a promising paradigm for training search agents to interact with external tools.<n>We propose textbfSearch textbfAgent textbfPolicy textbfOptimization (textbfSAPO), which stabilizes training via a conditional token-level KL constraint.
  arXiv  Detail & Related papers  (2026-03-10T04:07:39Z)
• Bayesian Conservative Policy Optimization (BCPO): A Novel Uncertainty-Calibrated Offline Reinforcement Learning with Credible Lower Bounds [1.2183405753834562]
  offline reinforcement learning (RL) aims to learn decision policies from a fixed batch of logged transitions.<n>We propose emphBayesian Conservative Policy Optimization (BCPO), a unified framework that converts uncertainty into emphprovably conservative policy improvement.<n>BCPO maintains a hierarchical Bayesian posterior over environment/value models, constructs a emphcredible lower bound (LCB) on action values, and performs policy updates under explicit KL regularization.
  arXiv  Detail & Related papers  (2026-03-06T01:46:02Z)
• Entropy Ratio Clipping as a Soft Global Constraint for Stable Reinforcement Learning [49.92803982100042]
  We propose using the entropy ratio between the current and previous policies as a new global metric.<n>We introduce an textbfEntropy Ratio (ERC) mechanism that imposes bidirectional constraints on the entropy ratio.<n>This stabilizes policy updates at the global distribution level and compensates for the inability of PPO-clip to regulate probability shifts of un-sampled actions.
  arXiv  Detail & Related papers  (2025-12-05T10:26:32Z)
• On the Design of KL-Regularized Policy Gradient Algorithms for LLM Reasoning [59.11784194183928]
  Policy gradient algorithms have been successfully applied to enhance the reasoning capabilities of large language models (LLMs)<n>Regularized Policy Gradient (RPG) view shows that the widely-used $k_3$ penalty is exactly the unnormalized KL.<n>RPG-REINFORCE with RPG-Style Clip improves accuracy by up to $+6$ absolute percentage points over DAPO.
  arXiv  Detail & Related papers  (2025-05-23T06:01:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.