ESPO: Early-Stopping Proximal Policy Optimization

• URL: http://arxiv.org/abs/2605.29860v1
• Date: Thu, 28 May 2026 12:40:22 GMT
• Title: ESPO: Early-Stopping Proximal Policy Optimization
• Abstract summary: ESPO (Early-Stopping Proximal Policy Optimization) detects trajectory failure on-the-fly and terminates rollouts early.<n>On DeepSeek-R1-Distill-Qwen-7B trained for mathematical reasoning, ESPO surpasses PPO on AIME2024 (46.28% vs. 45.25%), AMC2023 (85.83% vs. 82.94%), and MATH-500 (87.42% vs. 85.43%)
• Score: 78.79610718910628
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: When a large language model under reinforcement learning commits a wrong reasoning step early in a trajectory, standard algorithms force it to keep generating until the maximum horizon, spending compute on tokens that never receive positive reward and polluting advantage estimates with post-failure noise. We propose ESPO (Early-Stopping Proximal Policy Optimization), which detects trajectory failure on-the-fly and terminates rollouts early. At each generation step, ESPO computes a surrogate regret using only the logits already computed during sampling, and terminates when the smoothed cumulative regret significantly exceeds its estimated values. Truncated trajectories are treated as absorbing failure states with a terminal reward, concentrating negative temporal-difference (TD) errors near the detected failure step without any additional reward model or human annotation. On DeepSeek-R1-Distill-Qwen-7B trained for mathematical reasoning, ESPO surpasses PPO on AIME~2024 (46.28% vs. 45.25%), AMC~2023 (85.83% vs. 82.94%), and MATH-500 (87.42% vs. 85.43%), while saving more than 20% rollout tokens cumulatively.

Related papers

• RynnValue: Scaling Robotic Value Foundation Models with Temporal Distance [84.26294415726723]
  General-purpose reward models are increasingly the bottleneck for scaling robot learning.<n>We introduce RynnValue, an open-source value foundation model for robotic manipulation.<n>RynnValue scales to over 7,000 hours and roughly 3M instruction-conditioned clips without preference or progress annotations.
  arXiv  Detail & Related papers  (2026-08-10T17:09:37Z)
• Doomed from the Start: Early Abort of LLM Agent Episodes via a Recall-Controlled Probe Cascade [14.63102636181275]
  Large language model (LLM) agents often waste inference compute by continuing multi-step trajectories that are already doomed to fail.<n>We study early failure prediction and inference-time early stopping for LLM agents using hidden-state probes.<n>We turn this signal into a recall-controlled abort cascade for reducing LLM agent inference costs.
  arXiv  Detail & Related papers  (2026-07-07T17:03:55Z)
• AEGIS: A Backup Reflex for Physical AI [0.0]
  We introduce AEGIS, a selective escalation method that uses a lightweight probe on a weak policy's frozen activations to detect high-risk steps.<n>When the probe flags a step, control switches to a stronger separate policy, but only for the steps that need it.<n>On LIBERO-Spatial, AEGIS recovers 10.1% of the trajectories the weak policy alone loses, versus 4.6% for budget-matched blind escalation and 5.1% for a random-trigger placebo.
  arXiv  Detail & Related papers  (2026-06-04T19:09:22Z)
• Learn Where Outcomes Diverge: Efficient VLA RL via Probabilistic Chunk Masking [5.238545250784642]
  This paper presents Probabilistic Chunk Masking (PCM), a drop-in modification to GRPO that allocates gradient to a small, probabilistically selected subset of chunks per trajectory.<n>On three LIBERO benchmarks, PCM matches the final success rate of standard GRPO while achieving 2.38 times wall-clock speedup, 4.8 times faster gradient updates, and 60% lower peak activation memory.
  arXiv  Detail & Related papers  (2026-05-15T16:33:59Z)
• LaTER: Efficient Test-Time Reasoning via Latent Exploration and Explicit Verification [35.08680804423239]
  Chain-of-thought (CoT) reasoning improves large language models (LLMs) on difficult tasks, but it also makes inference expensive.<n>We propose Latent-Then-Explicit Reasoning (LaTER), a two-stage paradigm that first performs bounded exploration in a continuous latent space.<n>LaTER projects final-layer hidden states back to the input embedding space, preserves the latent KV cache, and uses entropy and model-native stop-token probes to decide when to switch.
  arXiv  Detail & Related papers  (2026-05-08T06:23:58Z)
• FIPO: Eliciting Deep Reasoning with Future-KL Influenced Policy Optimization [84.58281577727566]
  We present Future-KL Influenced Policy Optimization (FIPO), a reinforcement learning algorithm designed to overcome reasoning bottlenecks in large language models.<n>FIPO addresses this by incorporating discounted future-KL divergence into the policy update, creating a dense advantage formulation that re-weights tokens based on their influence on subsequent trajectory behavior.<n> evaluated on Qwen2.5-32B, FIPO extends the average chain-of-thought length from roughly 4,000 to over 10,000 tokens and increases AIME 2024 Pass@1 accuracy from 50.0% to a peak of 58.0%.
  arXiv  Detail & Related papers  (2026-03-20T10:24:50Z)
• Save the Good Prefix: Precise Error Penalization via Process-Supervised RL to Enhance LLM Reasoning [59.76691952347156]
  Reinforcement learning (RL) has emerged as a powerful framework for improving the reasoning capabilities of large language models (LLMs)<n>Most existing RL approaches rely on sparse outcome rewards, which fail to credit correct intermediate steps in partially successful solutions.<n>We propose Verifiable Prefix Policy Optimization (VPPO), which uses PRMs only to localize the first error during RL.
  arXiv  Detail & Related papers  (2026-01-26T21:38:20Z)
• Correct, Concise and Complete: Multi-stage Training For Adaptive Reasoning [11.179446105672461]
  We propose a multi-stage efficient reasoning method that combines supervised fine-tuning and reinforcement learning.<n>Our approach reduces response length by an average of 28% for 8B models and 40% for 32B models.<n>It achieves a superior trade-off compared to more complex state-of-the-art efficient reasoning methods.
  arXiv  Detail & Related papers  (2026-01-06T12:31:51Z)
• ZIP-RC: Optimizing Test-Time Compute via Zero-Overhead Joint Reward-Cost Prediction [57.799425838564]
  We present ZIP-RC, an adaptive inference method that equips models with zero-overhead inference-time predictions of reward and cost.<n> ZIP-RC improves accuracy by up to 12% over majority voting at equal or lower average cost.
  arXiv  Detail & Related papers  (2025-12-01T09:44:31Z)
• Improving Prediction Certainty Estimation for Reliable Early Exiting via Null Space Projection [16.838728310658105]
  We propose a novel early exiting method based on the Certainty-Aware Probability (CAP) score.<n>We show that our method can achieve an average speed-up ratio of 2.19x across all tasks with negligible performance degradation.
  arXiv  Detail & Related papers  (2025-06-08T05:08:34Z)
• Uncertainty-Penalized Direct Preference Optimization [52.387088396044206]
  We develop a pessimistic framework for DPO by introducing preference uncertainty penalization schemes. The penalization serves as a correction to the loss which attenuates the loss gradient for uncertain samples. We show improved overall performance compared to vanilla DPO, as well as better completions on prompts from high-uncertainty chosen/rejected responses.
  arXiv  Detail & Related papers  (2024-10-26T14:24:37Z)
• Iterative Reasoning Preference Optimization [84.15992372132507]
  We develop an iterative approach to optimize the preference between generated Chain-of-Thought (CoT) candidates. We show reasoning improves across repeated iterations of this scheme. For example, we see a large improvement from 55.6% to 81.6% on GSM8K and an accuracy of 88.7% with majority voting out of 32 samples.
  arXiv  Detail & Related papers  (2024-04-30T17:28:05Z)
• A2P-MANN: Adaptive Attention Inference Hops Pruned Memory-Augmented Neural Networks [3.682712058535653]
  We propose an online adaptive approach called A2P-MANN to limit the number of required attention inference hops in memory-augmented neural networks. The technique results in elimination of a large number of unnecessary computations in extracting the correct answer. The efficacy of the technique is assessed by using the twenty question-answering (QA) tasks of bAbI dataset.
  arXiv  Detail & Related papers  (2021-01-24T12:02:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.