SRPO: Self-Reflective Policy Optimization for Long-Horizon Reasoning
- URL: http://arxiv.org/abs/2608.23493v1
- Date: Mon, 24 Aug 2026 16:55:09 GMT
- Title: SRPO: Self-Reflective Policy Optimization for Long-Horizon Reasoning
- Abstract summary: Self-reflection is a powerful mechanism for credit assignment in human learning.<n>We propose Self-Reflective Policy Optimization (SRPO), a framework that internalizes this capability.<n>We demonstrate that SRPO achieves state-of-the-art performance across mathematical reasoning and long-horizon agentic benchmarks.
- Score: 36.32246019820315
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Self-reflection is a powerful mechanism for credit assignment in human learning, converting sparse outcome feedback into actionable guidance. However, its potential for post-training Large Language Models (LLMs) remains underexplored. We propose Self-Reflective Policy Optimization (SRPO), a framework that internalizes this capability. SRPO enables LLMs to analyze their own completed trajectories, synthesize errors into concise "reflection patches," and use reflection-conditioned teacher scores on student on-policy rollouts as dense token-level training signals. This process effectively transforms sparse terminal supervision into dense, token-level learning signals without requiring external critics, separate reward models, or larger teacher models. We demonstrate that SRPO achieves state-of-the-art performance across mathematical reasoning and long-horizon agentic benchmarks with exceptional data efficiency. Using a Qwen3-8B base model, SRPO attains 73.3% on AIME'24 using only 8% (0.08x) of the training FLOPs required by scaled supervised fine-tuning, while significantly improving success rates on WebShop (64.7%), ALFWorld (76.8%), and SWE-Bench-Lite (31.2%). Code is available at https://github.com/Galleons2029/SRPO
Related papers
- Meta-Learned Reward Shaping for Reinforcement Learning from Human Feedback [0.0]
We introduce MeRLa (Meta-Learned Reward Shaping), a framework that meta-learns a task-aware shaping function.<n>The learned shaping produces a composite reward that preserves policy optimality.<n>MeRLa retains its benefits when combined with process-based and principled-based enhanced rewards.
arXiv Detail & Related papers (2026-07-28T02:09:26Z) - Towards Robust Reinforcement Learning for Small-Scale Language Model Agents [5.301609879131692]
In this paper, fifteen (model, corpus) configurations were trained using Proximal Policy Optimization (PPO)<n>The experiments included Pythia-70M, 160M, 410M and SmolLM2-135M, 360M on the TinyStories, CNN/DailyMail, and Wiki-text103 corpora.<n>The proposed system converged stably in all experiments and improved preference win rate over the SFT baseline in configurations with a fluent prior and an informative reward signal.
arXiv Detail & Related papers (2026-07-27T21:30:44Z) - Beyond Scalar Rewards by Internalizing Reasoning into Score Distributions [80.33276868889341]
Z-Reward decouples reasoning-heavy judgment from efficient reward deployment.<n>We show that Z-Reward can serve as a differentiable reward signal for text-to-image optimization.
arXiv Detail & Related papers (2026-06-08T06:20:12Z) - TIPS: Turn-Level Information-Potential Reward Shaping for Search-Augmented LLMs [29.64044161352673]
We introduce Turn-Level Information Potential Reward Shaping (TIPS)<n>TIPS is a framework that assigns dense, turn-level rewards to each reasoning + tool-call segment based on the increased likelihood of the correct answer under a teacher model.<n>TIPS consistently outperforms GRPO/PPO baselines and substantially improves training stability.
arXiv Detail & Related papers (2026-03-11T17:45:14Z) - AceGRPO: Adaptive Curriculum Enhanced Group Relative Policy Optimization for Autonomous Machine Learning Engineering [52.67783579040657]
AceGRPO is a machine learning system that prioritizes tasks at the agent's learning frontier to maximize learning efficiency.<n>Our trained Ace-30B model achieves a 100% valid submission rate on MLE-Bench-Lite, approaches the performance of proprietary frontier models, and outperforms larger open-source baselines.
arXiv Detail & Related papers (2026-02-08T10:55:03Z) - SRPO: Self-Referential Policy Optimization for Vision-Language-Action Models [42.89413870143421]
Vision-Language-Action (VLA) models excel in robotic manipulation but are constrained by their heavy reliance on expert demonstrations.<n>Current VLA-RL methods, including group-based optimization approaches, are crippled by severe reward sparsity.<n>We propose Self-Referential Policy Optimization (SRPO), a novel VLA-RL framework.
arXiv Detail & Related papers (2025-11-19T16:52:23Z) - Asymmetric Proximal Policy Optimization: mini-critics boost LLM reasoning [49.57517969069136]
We introduce Asymmetric Proximal Policy Optimization (AsyPPO), a simple and scalable framework that restores the critics role while remaining efficient in large-model settings.<n>AsyPPO employs a set of lightweight mini-critics, each trained on disjoint prompt shards.<n>It consistently improves learning stability and performance across multiple benchmarks over strong baselines.
arXiv Detail & Related papers (2025-10-02T04:24:27Z) - Agentic Reinforcement Learning with Implicit Step Rewards [92.26560379363492]
Large language models (LLMs) are increasingly developed as autonomous agents using reinforcement learning (agentic RL)<n>We introduce implicit step rewards for agentic RL (iStar), a general credit-assignment strategy that integrates seamlessly with standard RL algorithms.<n>We evaluate our method on three challenging agent benchmarks, including WebShop and VisualSokoban, as well as open-ended social interactions with unverifiable rewards in SOTOPIA.
arXiv Detail & Related papers (2025-09-23T16:15:42Z) - Pre-Trained Policy Discriminators are General Reward Models [81.3974586561645]
We propose a scalable pre-training method named Policy Discriminative Learning (POLAR)<n>POLAR trains a reward model (RM) to discern identical policies and discriminate different ones.<n> Empirical results show that POLAR substantially outperforms traditional non-pre-trained methods.
arXiv Detail & Related papers (2025-07-07T16:56:31Z) - Self-Improving Robust Preference Optimization [22.493029742076605]
Online and offline RLHF methods have been highly successful in aligning AI with human preferences.<n>We propose Self-Improving Robust Preference Optimization (SRPO), a practical and mathematically principled offline RLHF framework.<n>We show that SRPO can be efficiently optimized using standard supervised learning techniques at scale.
arXiv Detail & Related papers (2024-06-03T17:53:25Z) - Self-Play Preference Optimization for Language Model Alignment [75.83359213697854]
Recent advancements suggest that directly working with preference probabilities can yield a more accurate reflection of human preferences.
We propose a self-play-based method for language model alignment, which treats the problem as a constant-sum two-player game.
Our approach, dubbed Self-Play Preference Optimization (SPPO), utilizes iterative policy updates to provably approximate the Nash equilibrium.
arXiv Detail & Related papers (2024-05-01T17:59:20Z) - Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learning [55.96599486604344]
We introduce an approach aimed at enhancing the reasoning capabilities of Large Language Models (LLMs) through an iterative preference learning process.
We use Monte Carlo Tree Search (MCTS) to iteratively collect preference data, utilizing its look-ahead ability to break down instance-level rewards into more granular step-level signals.
The proposed algorithm employs Direct Preference Optimization (DPO) to update the LLM policy using this newly generated step-level preference data.
arXiv Detail & Related papers (2024-05-01T11:10:24Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.