Progress- and Reliability-Oriented Group Policy Optimization for Agentic Reinforcement Learning

• URL: http://arxiv.org/abs/2607.04242v1
• Date: Sun, 05 Jul 2026 11:41:46 GMT
• Title: Progress- and Reliability-Oriented Group Policy Optimization for Agentic Reinforcement Learning
• Abstract summary: We propose ProGPO, a learned-critic-free method for context-consistent step-level learning.<n>We evaluate ProGPO on two challenging agentic tasks, ALFWorld and WebShop, with Qwen2.5-1.5B-Instruct.
• Score: 5.952158244195666
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Group-based reinforcement learning (RL) has become an effective paradigm for improving large language model agents on long-horizon interactive tasks. To obtain finer-grained policy updates than trajectory-level optimization, recent work has moved toward step-level group-based RL, where intermediate steps are grouped and compared within a rollout batch. However, step-level advantage estimation is sensitive to how groups are formed: grouping by broad state keys improves coverage but may compare actions taken under different histories, while enforcing historical consistency yields fairer comparisons at the cost of fragmented groups and missing peer-comparison signal. In this paper, we propose ProGPO (Progress- and Reliability-Oriented Group Policy Optimization), a learned-critic-free method for context-consistent step-level learning. ProGPO keeps exact-prefix action comparison, and complements sparse peer comparisons with transition credit derived from rollout-based state potentials. To estimate these potentials reliably, ProGPO combines semantic expansion with inverse-variance fusion across history depths. We evaluate ProGPO on two challenging agentic tasks, ALFWorld and WebShop, with Qwen2.5-1.5B-Instruct. Results show that ProGPO improves over matched agentic RL baselines under comparable computational overhead, and additional Qwen2.5-3B-Instruct experiments further test the scalability of the proposed method.

Related papers

• GUPO: Gradient Uncertainty-aware Policy Optimization for Post-Training Large Language Models [44.63057841001313]
  Group Relative Policy Optimization (GRPO) has become a widely used approach for post-training Large Language Models (LLMs)<n>In GRPO, the group gradients induced by different queries within the same mini-batch are directly averaged to form the policy update.<n>Our empirical analysis suggests that group-gradient conflicts tend to be associated with less effective policy updates.
  arXiv  Detail & Related papers  (2026-08-18T06:22:55Z)
• Progress-conditioned Group Policy Optimization for Long-Horizon Agentic Tasks [17.832722956008656]
  Group-based policy optimization has been increasingly used to train large language model (LLM) agents from sparse outcome rewards.<n>We propose Progress-conditioned Group Policy Optimization (ProGPO), which uses first-visit observation coverage only when all samples in a group receive zero outcome reward.<n>Experiments on two challenging agentic benchmarks, ALFWorld and WebShop with Qwen2.5-1.5/7B-Instruct, show that ProGPO consistently improves over group-based baselines.
  arXiv  Detail & Related papers  (2026-07-22T04:01:56Z)
• RRPO: Reference-Relative Policy Optimization with Stratified Conditional Rollouts [84.20747697259326]
  Group Relative Policy Optimization (GRPO) has shown strong effectiveness in reinforcement learning from verifiable feedback.<n>We propose textbfReference-Relative Policy Optimization (RRPO), which generalizes GRPO by replacing direct correctness-based advantage construction with reference-relative contrastive comparisons.<n>We evaluate RRPO using anchor-based contrastive advantages throughout policy optimization, without relying on task ground-truth verifiers.
  arXiv  Detail & Related papers  (2026-07-20T19:40:25Z)
• Group-Graph Policy Optimization for Long-Horizon Agentic Reinforcement Learning [85.02566758103008]
  Group-Graph Policy Optimization (G2PO) is a novel group-based Reinforcement Learning algorithm tailored for multi-turn agentic tasks.<n>G2PO substantially outperforms state-of-the-art prompt-based and RL baselines, achieving remarkable success rate improvements of up to 22.2% over GRPO.
  arXiv  Detail & Related papers  (2026-06-22T08:12:47Z)
• AdaGRPO: A Capability-Aware Adaptive Enhancement for Flow-based GRPO [78.36537400975298]
  Group Relative Policy Optimization (GRPO) has demonstrated remarkable success in aligning text-to-image (T2I) flow models with human preferences.<n>We have identified that the learning loop of current flow-based GRPO is fundamentally decoupled from the learner's current capability.<n>We propose Adaptive GRPO (AdaGRPO), a novel capability-aware RL algorithm tailored for flow models.
  arXiv  Detail & Related papers  (2026-06-05T02:07:08Z)
• Balanced Aggregation: Understanding and Fixing Aggregation Bias in GRPO [70.38763678943648]
  Reinforcement learning with verifiable rewards (RLVR) has become a central paradigm for improving reasoning and code generation in large language models.<n>Standard GRPO uses sequence aggregation, while recent work has advocated token aggregation as a better alternative.<n>We show that these two rules induce different optimization biases; token aggregation introduces sign-length coupling, while sequence aggregation implicitly downweights longer responses.
  arXiv  Detail & Related papers  (2026-04-14T09:48:46Z)
• Hierarchy-of-Groups Policy Optimization for Long-Horizon Agentic Tasks [23.119173310662365]
  Group-based reinforcement learning (RL) has advanced the capabilities of large language models on long-horizon agentic tasks.<n>We find a key issue in estimating stepwise relative advantages, namely context inconsistency, where steps within the same group may differ in their historical contexts.<n>We propose HGPO, which assigns each step to multiple hierarchical groups according to the consistency of historical contexts.
  arXiv  Detail & Related papers  (2026-02-26T09:58:10Z)
• iGRPO: Self-Feedback-Driven LLM Reasoning [88.83313431248473]
  Large Language Models (LLMs) have shown promise in solving complex mathematical problems, yet they still fall short of producing accurate and consistent solutions.<n>We introduce Iterative Group Relative Policy Optimization (iGRPO), a two-stage extension of GRPO that adds dynamic self-conditioning through model-generated drafts.<n>Under matched rollout budgets, iGRPO consistently outperforms GRPO across base models.
  arXiv  Detail & Related papers  (2026-02-09T18:45:11Z)
• GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization [133.27496265096445]
  We show how to apply Group Relative Policy Optimization under multi-reward setting without examining its suitability.<n>We then introduce Group reward-Decoupled Normalization Policy Optimization (GDPO), a new policy optimization method to resolve these issues.<n>GDPO consistently outperforms GRPO, demonstrating its effectiveness and generalizability for multi-reward reinforcement learning optimization.
  arXiv  Detail & Related papers  (2026-01-08T18:59:24Z)
• Compose Your Policies! Improving Diffusion-based or Flow-based Robot Policies via Test-time Distribution-level Composition [52.232968183793986]
  General Policy Composition (GPC) is a training-free method that enhances performance by combining the distributional scores of multiple pre-trained policies.<n>GPC consistently improves performance and adaptability across a diverse set of tasks.
  arXiv  Detail & Related papers  (2025-10-01T16:05:53Z)
• GTPO: Trajectory-Based Policy Optimization in Large Language Models [42.60363805227946]
  Policy-based optimizations are widely adopted today for the training and alignment of language models.<n>In this paper, we reveal and analyze two major limitations of GRPO.<n>We introduce GTPO, which identifies conflict tokens, tokens appearing in the same position across completions with opposite rewards.
  arXiv  Detail & Related papers  (2025-08-05T08:15:01Z)
• Group-in-Group Policy Optimization for LLM Agent Training [17.243181792126563]
  Group-in-Group Policy Optimization (GiGPO) is a novel RL algorithm that achieves fine-grained credit assignment for LLM agents.<n>We evaluate GiGPO on challenging agent benchmarks, including ALFWorld and WebShop, as well as tool-integrated reasoning on search-augmented QA tasks.
  arXiv  Detail & Related papers  (2025-05-16T08:26:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.