Beyond Uniform Token-Level Trust Region in LLM Reinforcement Learning

• URL: http://arxiv.org/abs/2606.10968v2
• Date: Wed, 10 Jun 2026 17:30:04 GMT
• Title: Beyond Uniform Token-Level Trust Region in LLM Reinforcement Learning
• Abstract summary: We propose a token-level masking rule that aligns updates with a finite-horizon policy-improvement bound via two mechanisms.<n> CPPO enhances training stability and significantly improves reasoning accuracy across various model scales.
• Score: 14.631835952085263
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement learning with verifiable rewards (RLVR) has become standard for improving LLM reasoning. However, existing PPO-style trust-region mechanisms remain position-agnostic by enforcing uniform thresholds across all tokens independently. This pointwise treatment conflicts with autoregressive generation in two critical ways. First, uniform thresholds ignore autoregressive asymmetry. Early-stage deviations produce compounding sequence-level drift, causing static thresholds to under-regulate early divergence and excessively constrain late-stage exploration. Second, evaluating token-level divergence in isolation overlooks cumulative prefix drift, granting the same divergence allowance regardless of how far the conditioning history has already deviated from the rollout policy. To address this limitation, we propose CPPO (Cumulative Prefix-divergence Policy Optimization), a token-level masking rule that aligns updates with a finite-horizon policy-improvement bound via two coupled mechanisms. First, a position-weighted threshold imposes stricter limits at early positions whose effects persist longer, relaxing constraints for late-stage tokens. Second, a cumulative prefix budget tracks historical deviations, dynamically restricting further token-level deviation to prevent compounding errors along the prefix. Empirically, CPPO enhances training stability and significantly improves reasoning accuracy across various model scales.

Related papers

• One Step, One Lead: Mitigating Higher-Order Interference in Multi-Domain Reinforcement Learning via Cross-Step Control [62.82372407840088]
  We show that same-point domain gradients may remain nearly even when consecutive realized updates partially reverse one another in output space.<n>We propose OSOL, which designates a focus domain at each iteration, to rank token-level rebound risk.<n>OSOL reaches a domain-macro average of 0.4822, improving by 5.7% over the strongest compared baseline, without explicit higher-order differentiation.
  arXiv  Detail & Related papers  (2026-09-06T08:32:59Z)
• Predictive Divergence Masks for LLM RL [44.851813004352046]
  Reinforcement learning for large language models (LLMs) typically relies on trust-region masks to stabilize off-policy updates.<n>Recent work DPPO improves the proximity criterion by replacing PPO's ratio-based test with a probability divergence between the behavior and training policies.<n>We propose the predictive divergence mask, which asks whether the next policy-gradient step will increase or decrease the same divergence used by the trust region.
  arXiv  Detail & Related papers  (2026-07-12T17:20:44Z)
• Rethinking the Divergence Regularization in LLM RL [56.952725733148746]
  Reinforcement learning (RL) has become a key component of post-training large language models (LLMs)<n>We propose Divergence Regularized Policy Optimization (DRPO), which replaces the hard mask with a smooth advantage-weighted quadratic regularizer on policy shift.<n>DRPO preserves the same trust-region geometry as DPPO while inducing bounded, continuous gradient weights.
  arXiv  Detail & Related papers  (2026-06-08T17:58:23Z)
• FAIR-Calib: Frontier-Aware Instability-Reweighted Calibration for Post-Training Quantization of Diffusion Large Language Models [50.86041384789834]
  We propose a two-stage PTQ framework for dLLMs that prioritizes the protection of fragile frontier states without requiring end-to-end diffusion rollouts.<n>We show that FAIR-Calib consistently outperforms state-of-the-art baselines on LLaDA and Dream (W4A4), reducing frontier decision flips and suppressing post-commit mismatches across diverse benchmarks.
  arXiv  Detail & Related papers  (2026-06-04T08:00:51Z)
• One-Way Policy Optimization for Self-Evolving LLMs [63.8638342097375]
  Reinforcement Learning with Verifiable Rewards (RLVR) has become a promising paradigm for scaling reasoning capabilities of Large Language Models (LLMs)<n>We propose One-Way Policy Optimization (OWPO), a method based on the principle of decoupling optimization direction from update magnitude.<n> Experimental results demonstrate that OWPO outperforms strong baselines, including DAPO, OPD, and MOPD.
  arXiv  Detail & Related papers  (2026-05-21T08:25:27Z)
• Rethinking Importance Sampling in LLM Policy Optimization: A Cumulative Token Perspective [22.848847562976633]
  Existing methods face a fundamental bias-variance dilemma: token-level IS ratios, as adopted by PPO and GRPO, introduce bias by ignoring prefix state distribution mismatch.<n>We propose CTPO, which combines the cumulative token IS ratio with position-adaptive clipping that scales log-space clip bounds according to the natural $sqrtt$ growth of the cumulative log-ratio.
  arXiv  Detail & Related papers  (2026-05-08T06:35:02Z)
• ANO: A Principled Approach to Robust Policy Optimization [19.004647288017814]
  We introduce a principled design space for policy optimization, demonstrating that a robust estimator must inherently suppress outliers.<n>We derive Anchored Neighborhood Optimization (ANO), a novel method that seamlessly replaces hard clipping with a redescending gradient mechanism.<n>In continuous (MuJoCo) and discrete (Atari) control, ANO establishes a robust state-of-the-art, uniquely preventing policy collapse even under highly aggressive learning rates.
  arXiv  Detail & Related papers  (2026-05-04T08:15:52Z)
• Improving Search Agent with One Line of Code [68.58667107354253]
  Tool-based Agentic Reinforcement Learning (TARL) has emerged as a promising paradigm for training search agents to interact with external tools.<n>We propose textbfSearch textbfAgent textbfPolicy textbfOptimization (textbfSAPO), which stabilizes training via a conditional token-level KL constraint.
  arXiv  Detail & Related papers  (2026-03-10T04:07:39Z)
• Bayesian Conservative Policy Optimization (BCPO): A Novel Uncertainty-Calibrated Offline Reinforcement Learning with Credible Lower Bounds [1.2183405753834562]
  offline reinforcement learning (RL) aims to learn decision policies from a fixed batch of logged transitions.<n>We propose emphBayesian Conservative Policy Optimization (BCPO), a unified framework that converts uncertainty into emphprovably conservative policy improvement.<n>BCPO maintains a hierarchical Bayesian posterior over environment/value models, constructs a emphcredible lower bound (LCB) on action values, and performs policy updates under explicit KL regularization.
  arXiv  Detail & Related papers  (2026-03-06T01:46:02Z)
• Near-Constant Strong Violation and Last-Iterate Convergence for Online CMDPs via Decaying Safety Margins [31.581870065866568]
  We study safe online reinforcement learning in Constrained Markov Decision Processes (CMDPs) under strong regret and violation metrics.<n>Existing primal-dual methods that achieve sublinear strong reward regret incur growing strong constraint violation or are restricted to average-iterate convergence due to inherent oscillations.<n>We propose the Flexible safety Domain Optimization via Margin-regularized Exploration (FlexDOME) algorithm, the first to provably achieve near-constant $tildeO(1)$ strong constraint violation alongside sublinear strong regret and non-asymptotic last-iterate convergence.
  arXiv  Detail & Related papers  (2026-02-11T14:54:26Z)
• Rethinking the Trust Region in LLM Reinforcement Learning [72.25890308541334]
  Proximal Policy Optimization (PPO) serves as the de facto standard algorithm for Large Language Models (LLMs)<n>We propose Divergence Proximal Policy Optimization (DPPO), which substitutes clipping with a more principled constraint.<n>DPPO achieves superior training and efficiency compared to existing methods, offering a more robust foundation for RL-based fine-tuning.
  arXiv  Detail & Related papers  (2026-02-04T18:59:04Z)
• A Step Back: Prefix Importance Ratio Stabilizes Policy Optimization [58.116300485427764]
  Reinforcement learning post-training can elicit reasoning behaviors in large language models.<n> token-level correction often leads to unstable training dynamics when the degree of off-policyness is large.<n>We propose a simple yet effective objective, Minimum Prefix Ratio (MinPRO)
  arXiv  Detail & Related papers  (2026-01-30T08:47:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.