StepOPSD: Step-Aware Online Preference Distillation for Agent Reinforcement Learning

• URL: http://arxiv.org/abs/2605.27140v1
• Date: Tue, 26 May 2026 15:07:03 GMT
• Title: StepOPSD: Step-Aware Online Preference Distillation for Agent Reinforcement Learning
• Abstract summary: We present StepOPSD, a post-rollout preference self-distillation framework that takes the agent step as the unit of credit redistribution.<n>StepOPSD decomposes trajectories into action-centered step segments, rescoring them under hindsight-enriched teacher contexts.<n>Results reveal a consistent two-knob law: smaller _clip acts as a broadly stabilizing local trust region, whereas the optimal global mixing strength _mix remains task-dependent.
• Score: 6.365332042924078
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement learning for multi-turn agents suffers from a credit-assignment mismatch: rewards are sparse and trajectory-level, while success often hinges on a few local decisions. Existing online policy distillation (OPD) provides denser token-level supervision, but typically treats heterogeneous agent trajectories as monolithic strings rather than causal interaction units. We present StepOPSD, a post-rollout preference self-distillation framework that takes the agent step as the unit of credit redistribution. StepOPSD decomposes trajectories into action-centered step segments, rescoring them under hindsight-enriched teacher contexts and converting token-level log-probability gaps into sign-preserving advantage shaping with a normalized per-step credit budget before the GRPO update. Across ALFWorld and Search-QA with Qwen3-1.7B and Qwen2.5-3B-Instruct, StepOPSD attains best or second-best results on subsets most sensitive to local causal errors, including first-place performance on ALFWorld Heat (79.1%), PickTwo (95.0%), Search-QA TriviaQA (61.6%), and tied-best performance on HotpotQA (40.4%). The results further reveal a consistent two-knob law: smaller α_clip acts as a broadly stabilizing local trust region, whereas the optimal global mixing strength λ_mix remains task-dependent. These findings suggest that step-aware distillation is most useful when trajectory-level rewards are weakly aligned with the local action that determines downstream success.

Related papers

• Granularity-Adaptive Credit Assignment for Long-Horizon LLM Agent Reinforcement Learning [7.885980895039013]
  We argue that the right resolution is state-dependent, and propose GACA, a critic-free estimator whose granularity follows an uncertainty-based criticality proxy.<n>GACA scores every step by the negative log-likelihood its own rollout already records, then blends the two advantages with a per-step weight that grows with that score.<n>On ALFWorld and WebShop, GACA improves task success over GRPO and GiGPO at both 1.5B and 7B scales.
  arXiv  Detail & Related papers  (2026-09-11T04:19:44Z)
• MileGPO: Milestone Inference with Local Evidence for Graph-Based Policy Optimization of Long-Horizon LLM Agents [26.262385802144323]
  We propose MileGPO, which derives process-level credit from grouped on-policy rollouts through three designs.<n>MileGPO requires neither auxiliary models nor additional environment interaction.
  arXiv  Detail & Related papers  (2026-08-20T08:58:27Z)
• TRCA: Transition-wise Rubric Credit Assignment for Long-horizon LLM Agents [22.01777615106231]
  Long-horizon large language model (LLM) agents are typically optimized with sparse terminal outcomes.<n>We propose Transition-wise Credit Assignment, which derives step-level supervision directly from action-induced transitions.<n>Experiments on ALFWorld, WebShop, and seven search-augmented question-answering benchmarks show consistent improvements over the evaluated baselines.
  arXiv  Detail & Related papers  (2026-08-17T06:19:12Z)
• AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning [58.76655851910778]
  AgentOPSD is a critic-free, recursive method for turn-level credit assignment in agentic reinforcement learning.<n>We evaluate AgentOPSD on ALFWorld, WebShop, and Search-QA using Qwen2.5 models at two scales (3B and 7B)
  arXiv  Detail & Related papers  (2026-08-06T13:00:59Z)
• ARCO: Adaptive Rubric with Co-Evolution for Multi-Step LLM-Based Agents [12.091072999307622]
  ARCO is an adaptive rubric framework for reinforcement learning.<n>It improves the best EM in every setting over strong outcome-, rubric-, and process-reward baselines.<n>Its rubrics are step-specific, robust to design choices, and useful for diagnosing agent behavior.
  arXiv  Detail & Related papers  (2026-06-19T09:38:02Z)
• TRACE: A Unified Rollout Budget Allocation Framework for Efficient Agentic Reinforcement Learning [52.67756371231985]
  Reinforcement learning with verifiable rewards (RLVR) is a promising approach for enhancing reasoning and agentic behavior in large language models.<n>We introduce Tree Rollout Allocation for Contrastive Exploration (TRACE), a unified rollout allocation framework.<n>Technically, TRACE allocates rollout budget to both prompt roots and intermediate prefixes that are most likely to yield mixed terminal rewards.
  arXiv  Detail & Related papers  (2026-06-09T17:16:03Z)
• VeriGate: Verifier-Gated Step-Level Supervision for GRPO [51.26100506256885]
  Group Relative Policy Optimization is an effective recipe for training reasoning models with verifier-based outcome rewards.<n>We present VeriGate, a verifier-gated extension of GRPO that addresses these limitations with three design choices.<n>We show that VeriGate improves average accuracy by about 20% and 12% for 1.5B and 7B models respectively.
  arXiv  Detail & Related papers  (2026-05-28T18:20:32Z)
• PAIR: Prefix-Aware Internal Reward Model for Multi-Turn Agent Optimization [20.910779784518798]
  Group Relative Policy Optimization (GRPO) has been emerging as a leading choice, but its reliance on sparse outcome rewards severely limits credit assignment across intermediate steps.<n>We propose a two-stage model with a frozen hidden-state probe estimating belief-consistency and a lightweight attention-based head correcting it toward grounded correctness.<n> Experimental results show that PAIR achieves the highest AUROC on contaminated trajectories while operating at negligible inference cost.
  arXiv  Detail & Related papers  (2026-05-18T05:39:30Z)
• PiCA: Pivot-Based Credit Assignment for Search Agentic Reinforcement Learning [11.926589875842359]
  Large Language Model (LLM)-based search agents have significantly improved the performance of knowledge-intensive tasks.<n>Existing methods encounter critical challenges in long-horizon credit assignment.<n>We propose Pivot-Based Credit Assignment (PiCA) to reformulate the search trajectory as a sequential process of cumulative search progress.
  arXiv  Detail & Related papers  (2026-05-10T03:21:47Z)
• Improving Search Agent with One Line of Code [68.58667107354253]
  Tool-based Agentic Reinforcement Learning (TARL) has emerged as a promising paradigm for training search agents to interact with external tools.<n>We propose textbfSearch textbfAgent textbfPolicy textbfOptimization (textbfSAPO), which stabilizes training via a conditional token-level KL constraint.
  arXiv  Detail & Related papers  (2026-03-10T04:07:39Z)
• Verified Critical Step Optimization for LLM Agents [67.05296684575445]
  Critical Step Optimization focuses preference learning on verified critical steps.<n>Method starts from failed policy trajectories rather than expert demonstrations.<n>Experiments on GAIA-Text-103 and XBench-DeepSearch show that CSO achieves 37% and 26% relative improvement over the SFT baseline.
  arXiv  Detail & Related papers  (2026-02-03T11:41:02Z)
• Prune-Then-Plan: Step-Level Calibration for Stable Frontier Exploration in Embodied Question Answering [52.69447404069251]
  Large vision-language models (VLMs) have improved embodied question answering (EQA) agents by providing strong semantic priors for open-vocabulary reasoning.<n>We propose Prune-Then-Plan, a framework that stabilizes exploration through step-level calibration.
  arXiv  Detail & Related papers  (2025-11-24T22:50:50Z)
• Agentic Reinforcement Learning with Implicit Step Rewards [92.26560379363492]
  Large language models (LLMs) are increasingly developed as autonomous agents using reinforcement learning (agentic RL)<n>We introduce implicit step rewards for agentic RL (iStar), a general credit-assignment strategy that integrates seamlessly with standard RL algorithms.<n>We evaluate our method on three challenging agent benchmarks, including WebShop and VisualSokoban, as well as open-ended social interactions with unverifiable rewards in SOTOPIA.
  arXiv  Detail & Related papers  (2025-09-23T16:15:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.