APPO: Agentic Procedural Policy Optimization

• URL: http://arxiv.org/abs/2606.12384v1
• Date: Wed, 10 Jun 2026 17:47:07 GMT
• Title: APPO: Agentic Procedural Policy Optimization
• Abstract summary: We study agentic RL from two perspectives: textitwhere to branch and how to assign credit after branching<n>We propose APPO, which shifts branching and credit assignment from coarse interaction units to fine-grained decision points in the sequence.<n>APPO consistently improves strong agentic RL baselines by nearly 4 points, while keeping efficient tool-calls and maintaining behavior interpretability.
• Score: 28.730334202611257
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advances in agentic Reinforcement Learning (RL) have substantially improved the multi-turn tool-use capabilities of large language model agents. However, most existing methods assign credit over coarse heuristic units, such as tool-call boundaries or fixed workflows, making it difficult to identify which intermediate decisions influence downstream outcomes. In this work, we study agentic RL from two perspectives: \textit{where to branch and how to assign credit after branching}. Our pilot analysis shows that influential decision points are broadly distributed throughout the generated sequence rather than concentrated at tool calls, while token entropy alone does not reliably reflect their impact on final outcomes. Motivated by these observations, we propose \textbf{Agentic Procedural Policy Optimization (APPO)}, which shifts branching and credit assignment from coarse interaction units to fine-grained decision points in the sequence. APPO selects branching locations using a Branching Score that combines token uncertainty with policy-induced likelihood gains of subsequent continuations, enabling more targeted exploration while filtering out spurious high-entropy positions. It further introduces procedure-level advantage scaling to better distribute credit across branched rollouts. Experiments on 13 benchmarks show that APPO consistently improves strong agentic RL baselines by nearly 4 points, while keeping efficient tool-calls and maintaining behavior interpretability.

Related papers

• Reconciling Process Supervision with Outcome-Based Credit in Agentic Policy Optimization [13.354353071601379]
  We introduce TASPO, which converts privileged supervision into outcome-grounded action credit.<n>Across three agentic benchmarks, TASPO improves over GRPO by 10.6% and generalizes better to unseen tasks.
  arXiv  Detail & Related papers  (2026-08-31T16:51:50Z)
• IAPO: Influence-Aware Policy Optimization for Credit Assignment in Multi-Turn Service Agents [10.577620258047665]
  This work advances the understanding of credit assignment in multi-turn user interactions.<n>It provides a principled approach to training service agents from sparse outcome feedback.
  arXiv  Detail & Related papers  (2026-08-25T14:14:36Z)
• Contrastive Branch Policy Optimization [16.16893757028268]
  Reinforcement learning with verifiable rewards (RLVR) enables language models to learn multi-turn interaction with external tools.<n>Existing methods tend to conflate two distinct problems: allocating a fixed rollout budget and translating branch outcomes into token-level credit.<n>We introduce Contrastive Branch Policy Optimization (CBPO), which disentangles these two problems and assigns a dedicated mechanism to each.
  arXiv  Detail & Related papers  (2026-08-25T09:25:55Z)
• AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning [58.76655851910778]
  AgentOPSD is a critic-free, recursive method for turn-level credit assignment in agentic reinforcement learning.<n>We evaluate AgentOPSD on ALFWorld, WebShop, and Search-QA using Qwen2.5 models at two scales (3B and 7B)
  arXiv  Detail & Related papers  (2026-08-06T13:00:59Z)
• TRACE: A Unified Rollout Budget Allocation Framework for Efficient Agentic Reinforcement Learning [52.67756371231985]
  Reinforcement learning with verifiable rewards (RLVR) is a promising approach for enhancing reasoning and agentic behavior in large language models.<n>We introduce Tree Rollout Allocation for Contrastive Exploration (TRACE), a unified rollout allocation framework.<n>Technically, TRACE allocates rollout budget to both prompt roots and intermediate prefixes that are most likely to yield mixed terminal rewards.
  arXiv  Detail & Related papers  (2026-06-09T17:16:03Z)
• TAPO: Tool-Aware Policy Optimization via Credit Transfer for Multimodal Search Agents [54.08846865906602]
  We characterize credit misassignment as a systematic failure mode of GRPO in tool-augmented multimodal search agents.<n>We propose Tool-Aware Policy Optimization (TAPO), which exploits the parameter-determinism property of information-acquisition tools.
  arXiv  Detail & Related papers  (2026-06-04T07:15:43Z)
• Reducing Credit Assignment Variance via Counterfactual Reasoning Paths [11.514388061694268]
  Reinforcement learning for multi-step reasoning with large language models (LLMs) often relies on sparse terminal rewards.<n>We introduce a counterfactual comparison-based credit assignment framework, which samples multiple reasoning trajectories under the same input.<n>We construct an implicit process-level advantage estimator that transforms sparse terminal rewards into step-sensitive learning signals.
  arXiv  Detail & Related papers  (2026-04-20T13:33:54Z)
• HISR: Hindsight Information Modulated Segmental Process Rewards For Multi-turn Agentic Reinforcement Learning [47.756127177035246]
  We propose (HISR) exploiting Hindsight Information to modulate Segmental process Rewards.<n>Specifically, a segment-level process RM is presented to assign rewards for each sub-goal in the task.<n>To emphasize significant segments in the trajectory, a hindsight model is devised to reflect the preference of performing a certain action after knowing the trajectory outcome.
  arXiv  Detail & Related papers  (2026-03-19T09:42:05Z)
• Evaluate-as-Action: Self-Evaluated Process Rewards for Retrieval-Augmented Agents [24.080021799509847]
  textscEvalAct (Evaluate-as-Action) converts implicit retrieval quality assessment into an explicit action.<n>textscEvalAct achieves the best average accuracy, with the largest gains on multi-hop tasks.
  arXiv  Detail & Related papers  (2026-03-10T05:22:40Z)
• DLLM Agent: See Farther, Run Faster [94.74432470237817]
  Diffusion large language models (DLLMs) have emerged as an alternative to autoregressive (AR) decoding with appealing efficiency and modeling properties.<n>We study this in a controlled setting by instantiatingDLLM and AR backbones within the same agent workflow.<n>We find thatDLLM Agents are on average over 30% faster end to end than AR agents, with some cases exceeding 8x speedup.
  arXiv  Detail & Related papers  (2026-02-07T09:01:18Z)
• Training Multi-Turn Search Agent via Contrastive Dynamic Branch Sampling [29.182538022605627]
  Branching Relative Policy Optimization (BranPO) is a value-free method that provides step-level contrastive supervision without dense rewards.<n>BranPO truncates trajectories near the tail and resamples alternative continuations to construct contrastive suffixes over shared prefixes.<n>To further boost efficiency and stabilize training, we introduce difficulty-aware branch sampling to adapt branching frequency across tasks, and redundant step masking to suppress uninformative actions.
  arXiv  Detail & Related papers  (2026-02-03T16:43:09Z)
• Agentic Reinforcement Learning with Implicit Step Rewards [92.26560379363492]
  Large language models (LLMs) are increasingly developed as autonomous agents using reinforcement learning (agentic RL)<n>We introduce implicit step rewards for agentic RL (iStar), a general credit-assignment strategy that integrates seamlessly with standard RL algorithms.<n>We evaluate our method on three challenging agent benchmarks, including WebShop and VisualSokoban, as well as open-ended social interactions with unverifiable rewards in SOTOPIA.
  arXiv  Detail & Related papers  (2025-09-23T16:15:42Z)
• Agentic Reinforced Policy Optimization [66.96989268893932]
  Large-scale reinforcement learning with verifiable rewards (RLVR) has demonstrated its effectiveness in harnessing the potential of large language models (LLMs) for single-turn reasoning tasks.<n>Current RL algorithms inadequately balance the models' intrinsic long-horizon reasoning capabilities and their proficiency in multi-turn tool interactions.<n>We propose Agentic Reinforced Policy Optimization (ARPO), a novel agentic RL algorithm tailored for training multi-turn LLM-based agents.
  arXiv  Detail & Related papers  (2025-07-26T07:53:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.