Reusing Rollouts under Policy Lag: Prefix-Normalized Policy Optimization for LLM Reinforcement Learning

• URL: http://arxiv.org/abs/2608.01418v1
• Date: Sun, 02 Aug 2026 18:02:03 GMT
• Title: Reusing Rollouts under Policy Lag: Prefix-Normalized Policy Optimization for LLM Reinforcement Learning
• Abstract summary: Autoregressive rollout generation is a major computational cost in reinforcement learning.<n>PNPO replaces the cumulative ratio with the mean of likelihood ratios along each causal prefix.<n>PNPO does not consistently outperform GSPO with one epoch.<n>Four-epoch PNPO reaches a final macro Avg@32 of 49.66 after 150 rollout batches, comparable to the 49.56 reached after 600 batches with one epoch.
• Score: 21.43528083212271
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Autoregressive rollout generation is a major computational cost in reinforcement learning for large language models. Reusing each rollout batch for additional learner updates amortizes this cost, but later updates become increasingly off-policy as the learner departs from the behavior policy. At a token position, exact off-policy correction must account for both the current action and the probability of reaching its prefix. The cumulative importance ratio provides this correction, but its product form can produce an unwieldy dynamic range. We study Prefix-Normalized Policy Optimization (PNPO), which replaces the cumulative ratio with the geometric mean of likelihood ratios along each causal prefix, preserving causal-prefix dependence at each position while compressing the log-weight scale. In controlled long-context mathematical reasoning experiments, we induce two off-policy regimes by using one or four policy-update epochs per rollout batch. PNPO does not consistently outperform GSPO with one epoch. With four epochs, it attains the highest observed Avg@32 on each benchmark; the unweighted mean of the three independently selected benchmark peaks is 50.24, 3.00 percentage points above GSPO. Under a matched 2,400-update budget, four-epoch PNPO reaches a final macro Avg@32 of 49.66 after 150 rollout batches, comparable to the 49.56 reached after 600 batches with one epoch. These results provide preliminary evidence that PNPO can be advantageous as training moves further off-policy.

Related papers

• Why Sample What You Can Enumerate? Exact Policy Optimization for Genomic Tool Selection [23.92446384027772]
  Reinforcement learning over a frozen reasoner has become a common recipe for teaching a policy which external tools invoke.<n>We show that this recipe becomes structurally mismatched in specialist scientific settings where the complete tool-subset space is enumerable.<n>We introduce FGPO, which scores every tool subset and optimize the exact action expectation, so each update sees the complete action space.
  arXiv  Detail & Related papers  (2026-09-09T14:20:47Z)
• CliffRank: A Dual-Branch Framework for Activity-Cliff Ranking Prediction [8.452103801866203]
  CliffRank trains two parallel predictors with mean squared error, a thresholded listwise loss, and Pairwise Preference Consistency (PPC)<n>On three antimicrobial peptide datasets, CliffRank with ESM2-t12 achieved the highest mean Spearman correlation of 0.5393 and mean Recall@50 of 21.4.<n>Future work should evaluate more targets and antimicrobial peptide systems, develop adaptive PPC schedules, and incorporate protein or membrane context when available.
  arXiv  Detail & Related papers  (2026-09-01T07:57:56Z)
• Physics-Guided Policy Optimization with Self-Distillation [13.212487988108592]
  Self-distilled policy optimization (SDPO) has become a popular paradigm for LLM post-training.<n>We propose Physics-Guided Policy Optimization (PGPO), which introduces an information-modulated step-size multiplier.<n>We show that this modulation preserves the order-1 weak-approximation guarantees of vanilla SGD, and incurs negligible overhead per iteration.
  arXiv  Detail & Related papers  (2026-06-02T13:20:39Z)
• Trust the Batch, On- or Off-Policy: Adaptive Policy Optimization for RL Post-Training [50.86545293331458]
  Reinforcement learning is structurally harder than supervised learning because the policy changes the data distribution it learns from.<n>We present a simple yet effective batch-adaptive objective that replaces fixed clipping with the normalized effective sample size of the policy ratios.
  arXiv  Detail & Related papers  (2026-05-12T16:44:47Z)
• Rethinking Importance Sampling in LLM Policy Optimization: A Cumulative Token Perspective [22.848847562976633]
  Existing methods face a fundamental bias-variance dilemma: token-level IS ratios, as adopted by PPO and GRPO, introduce bias by ignoring prefix state distribution mismatch.<n>We propose CTPO, which combines the cumulative token IS ratio with position-adaptive clipping that scales log-space clip bounds according to the natural $sqrtt$ growth of the cumulative log-ratio.
  arXiv  Detail & Related papers  (2026-05-08T06:35:02Z)
• FIPO: Eliciting Deep Reasoning with Future-KL Influenced Policy Optimization [84.58281577727566]
  We present Future-KL Influenced Policy Optimization (FIPO), a reinforcement learning algorithm designed to overcome reasoning bottlenecks in large language models.<n>FIPO addresses this by incorporating discounted future-KL divergence into the policy update, creating a dense advantage formulation that re-weights tokens based on their influence on subsequent trajectory behavior.<n> evaluated on Qwen2.5-32B, FIPO extends the average chain-of-thought length from roughly 4,000 to over 10,000 tokens and increases AIME 2024 Pass@1 accuracy from 50.0% to a peak of 58.0%.
  arXiv  Detail & Related papers  (2026-03-20T10:24:50Z)
• Rethinking the Trust Region in LLM Reinforcement Learning [72.25890308541334]
  Proximal Policy Optimization (PPO) serves as the de facto standard algorithm for Large Language Models (LLMs)<n>We propose Divergence Proximal Policy Optimization (DPPO), which substitutes clipping with a more principled constraint.<n>DPPO achieves superior training and efficiency compared to existing methods, offering a more robust foundation for RL-based fine-tuning.
  arXiv  Detail & Related papers  (2026-02-04T18:59:04Z)
• GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization [133.27496265096445]
  We show how to apply Group Relative Policy Optimization under multi-reward setting without examining its suitability.<n>We then introduce Group reward-Decoupled Normalization Policy Optimization (GDPO), a new policy optimization method to resolve these issues.<n>GDPO consistently outperforms GRPO, demonstrating its effectiveness and generalizability for multi-reward reinforcement learning optimization.
  arXiv  Detail & Related papers  (2026-01-08T18:59:24Z)
• A-3PO: Accelerating Asynchronous LLM Training with Staleness-aware Proximal Policy Approximation [2.5291809836356998]
  We introduce a proximal policy to decouple the off-policy corrections from the controlling policy updates.<n>The policy requires an extra forward pass through the network at each training step, creating a computational bottleneck.<n>We observe that since the proximal policy only serves as a trust region anchor between the behavior and target policies, we can approximate it through simple without explicit computation.
  arXiv  Detail & Related papers  (2025-12-06T19:37:39Z)
• Single-stream Policy Optimization [21.214853668053234]
  We introduce Single-stream Policy Optimization (SPO)<n>SPO replaces per-group baselines with a persistent, KL-adaptive value tracker and normalizes advantages globally across the batch.<n>Being group-free, SPO enables higher throughput and scales effectively in long-horizon or tool-integrated settings.
  arXiv  Detail & Related papers  (2025-09-16T16:39:11Z)
• Squeeze the Soaked Sponge: Efficient Off-policy Reinforcement Finetuning for Large Language Model [56.92219181993453]
  We propose Reincarnating Mix-policy Proximal Policy Gradient (ReMix) to enable on-policyRFT methods like PPO and GRPO to leverage off-policy data.<n>ReMix consists of three major components: (1) Mix-policy proximal policy gradient with an increased Update-To-Data (UTD) ratio for efficient training; (2) KL-Convex policy constraint to balance the trade-off between stability and flexibility; (3) Policy reincarnation to achieve a seamless transition from efficient early-stage learning to steady improvement.
  arXiv  Detail & Related papers  (2025-07-09T14:29:45Z)
• You May Not Need Ratio Clipping in PPO [117.03368180633463]
  Proximal Policy Optimization (PPO) methods learn a policy by iteratively performing multiple mini-batch optimization epochs of a surrogate objective with one set of sampled data. Ratio clipping PPO is a popular variant that clips the probability ratios between the target policy and the policy used to collect samples. We show in this paper that such ratio clipping may not be a good option as it can fail to effectively bound the ratios. We show that ESPO can be easily scaled up to distributed training with many workers, delivering strong performance as well.
  arXiv  Detail & Related papers  (2022-01-31T20:26:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.