PGPO: Potential-Guided Policy Optimization for Multi-Turn Agentic Tasks

• URL: http://arxiv.org/abs/2609.02236v1
• Date: Wed, 02 Sep 2026 07:44:16 GMT
• Title: PGPO: Potential-Guided Policy Optimization for Multi-Turn Agentic Tasks
• Abstract summary: We propose Potential-Guided Policy Optimization for multi-turn agentic tasks.<n>It estimates empirical state potentials from anchor-state-group return statistics within each rollout group.<n>It then derives action advantages from potential differences between adjacent states, enabling cross-trajectory credit propagation.
• Score: 41.82007055765541
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Group-based reinforcement learning (RL) has become an effective paradigm for LLM post-training, but in multi-turn agentic tasks with sparse terminal rewards, it often provides coarse credit for intermediate actions. To obtain more fine-grained credit assignment, recent work such as GiGPO introduces step-level advantages for intermediate actions. However, these step-level signals still rely on the final outcome of each individual trajectory. As a result, actions within failed trajectories can remain poorly differentiated, so effective actions can receive the same unfavorable credit as erroneous ones. In this work, we propose Potential-Guided Policy Optimization (PGPO) for multi-turn agentic tasks. PGPO estimates empirical state potentials from anchor-state-group return statistics within each rollout group. It then derives action advantages from potential differences between adjacent states, enabling cross-trajectory credit propagation. This provides finer-grained step-level credit assignment, especially within failed trajectories. Experiments on ALFWorld and WebShop show strong overall performance relative to recent group-based RL methods. Further analysis provides evidence that PGPO yields more informative failure-side credit signals with negligible training overhead.

Related papers

• Progress-conditioned Group Policy Optimization for Long-Horizon Agentic Tasks [17.832722956008656]
  Group-based policy optimization has been increasingly used to train large language model (LLM) agents from sparse outcome rewards.<n>We propose Progress-conditioned Group Policy Optimization (ProGPO), which uses first-visit observation coverage only when all samples in a group receive zero outcome reward.<n>Experiments on two challenging agentic benchmarks, ALFWorld and WebShop with Qwen2.5-1.5/7B-Instruct, show that ProGPO consistently improves over group-based baselines.
  arXiv  Detail & Related papers  (2026-07-22T04:01:56Z)
• Group-Graph Policy Optimization for Long-Horizon Agentic Reinforcement Learning [85.02566758103008]
  Group-Graph Policy Optimization (G2PO) is a novel group-based Reinforcement Learning algorithm tailored for multi-turn agentic tasks.<n>G2PO substantially outperforms state-of-the-art prompt-based and RL baselines, achieving remarkable success rate improvements of up to 22.2% over GRPO.
  arXiv  Detail & Related papers  (2026-06-22T08:12:47Z)
• TRACE: A Unified Rollout Budget Allocation Framework for Efficient Agentic Reinforcement Learning [52.67756371231985]
  Reinforcement learning with verifiable rewards (RLVR) is a promising approach for enhancing reasoning and agentic behavior in large language models.<n>We introduce Tree Rollout Allocation for Contrastive Exploration (TRACE), a unified rollout allocation framework.<n>Technically, TRACE allocates rollout budget to both prompt roots and intermediate prefixes that are most likely to yield mixed terminal rewards.
  arXiv  Detail & Related papers  (2026-06-09T17:16:03Z)
• When Denser Credit Is Not Enough: Evidence-Calibrated Policy Optimization for Long-Horizon LLM Agent Training [8.367896852036699]
  We propose Evidence-Calibrated Policy Optimization (ECPO), a critic-free policy optimization algorithm that calibrates step-level credit before policy updates.<n>ECPO consistently outperforms strong baselines, improving GiGPO by +5.2/+7.3 success points on ALFWorld/WebShop with Qwen2.5-1.5B while adding only 0.1% additional advantage-computation overhead.
  arXiv  Detail & Related papers  (2026-06-04T08:54:09Z)
• GAGPO: Generalized Advantage Grouped Policy Optimization [13.085463559931156]
  We propose a critic-free reinforcement learning method for precise, step-aligned temporal credit assignment.<n>Experiments on ALFWorld and WebShop show that GAGPO outperforms strong reinforcement learning baselines.
  arXiv  Detail & Related papers  (2026-05-13T09:10:03Z)
• Enhancing LLM-based Search Agents via Contribution Weighted Group Relative Policy Optimization [47.7937991619078]
  We propose Contribution-Weighted GRPO, a framework that integrates process supervision into group relative policy optimization.<n>CW-GRPO employs an LLM judge to assess the retrieval utility and reasoning correctness at each search round, producing per-round contribution scores.<n> Experiments on multiple knowledge-intensive benchmarks show that CW-GRPO outperforms standard GRPO by 5.0% on Qwen3-8B and 6.3% on Qwen3-1.7B.
  arXiv  Detail & Related papers  (2026-04-15T17:37:59Z)
• Information Gain-based Policy Optimization: A Simple and Effective Approach for Multi-Turn LLM Agents [28.145430029174577]
  Large language model (LLM)-based agents are increasingly trained with reinforcement learning (RL) to enhance their ability to interact with external environments.<n>Existing approaches typically rely on outcome-based rewards that are only provided at the final answer.<n>In this paper, we propose Information Gain-based Policy Optimization (IGPO), a simple yet effective RL framework that provides dense and intrinsic supervision for multi-turn agent training.
  arXiv  Detail & Related papers  (2025-10-16T17:59:32Z)
• Agentic Reinforcement Learning with Implicit Step Rewards [92.26560379363492]
  Large language models (LLMs) are increasingly developed as autonomous agents using reinforcement learning (agentic RL)<n>We introduce implicit step rewards for agentic RL (iStar), a general credit-assignment strategy that integrates seamlessly with standard RL algorithms.<n>We evaluate our method on three challenging agent benchmarks, including WebShop and VisualSokoban, as well as open-ended social interactions with unverifiable rewards in SOTOPIA.
  arXiv  Detail & Related papers  (2025-09-23T16:15:42Z)
• From Novice to Expert: LLM Agent Policy Optimization via Step-wise Reinforcement Learning [62.54484062185869]
  We introduce StepAgent, which utilizes step-wise reward to optimize the agent's reinforcement learning process.<n>We propose implicit-reward and inverse reinforcement learning techniques to facilitate agent reflection and policy adjustment.
  arXiv  Detail & Related papers  (2024-11-06T10:35:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.