Beyond Outcome Rewards: Step-Level Self-Distilled Policy Optimization for Deep Search Agents

• URL: http://arxiv.org/abs/2608.12764v1
• Date: Thu, 13 Aug 2026 03:13:18 GMT
• Title: Beyond Outcome Rewards: Step-Level Self-Distilled Policy Optimization for Deep Search Agents
• Abstract summary: Deep search agents operate over trajectories spanning dozens of steps.<n>On-policy self-distillation (OPSD) addresses this by using the model's own logits as dense token-level teachers.<n>We propose Step-Level Self-Distilled Policy Optimization (SSPO), which converts teacher-student disagreement into step-level advantage weights.
• Score: 15.62688304085298
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Deep search agents operate over trajectories spanning dozens of steps, yet standard reinforcement learning provides only a single outcome reward per trajectory, which is far too sparse for effective credit assignment. On-policy self-distillation (OPSD) addresses this by using the model's own logits as dense token-level teachers, but extending it to search agents introduces a fundamental tension: the teacher, having access to privileged information such as the correct answer, produces a distribution that differs systematically from the student's exploration-based reasoning, and naive distillation causes the student to inherit this information asymmetry rather than learn better search strategies. We resolve this tension through two contributions. First, we construct Evidence Anchors, which are concise, step-level evidence snippets extracted from the web, as privileged information that captures key reasoning steps without revealing the entire answer path. Second, we propose Step-Level Self-Distilled Policy Optimization (SSPO), which converts teacher-student disagreement into step-level advantage weights within GRPO, applied exclusively to incorrect trajectories. This design decouples what to update from how much to update: the outcome reward determines the direction of policy change, while the teacher modulates its magnitude at each step. Correct trajectories are left untouched, preserving their diversity. On Qwen3-8B, SSPO consistently outperforms GRPO across BrowseComp, GAIA, and FRAMES, surpassing or matching GRPO trained with twice as many gradient steps while adding only about 5 percent overhead per step from a single additional forward pass.

Related papers

• AHEAD: Adaptive Hindsight with Environment-Augmented Distillation for Agentic RL [14.402945966056764]
  Self-distillation methods can provide finer-grained supervision by augmenting RL with privileged information.<n>We propose AHEAD, a step-aware framework that matches different supervision sources to different step types.<n>AHEAD raises task success (+13.3 points on ALFWorld), reaches a given success rate in fewer training steps, and solves tasks within tighter interaction budgets than outcome-only RL and prior self-distillation baselines.
  arXiv  Detail & Related papers  (2026-08-25T06:23:48Z)
• Latent On-Policy Self-Distillation [55.57800223145407]
  On-policy self-distillation (OPSD) offers an effective pathway by using a privileged self-teacher to provide dense supervision on the student's own trajectories.<n>Existing methods still rely heavily on designer-specified privileged artifacts.<n>We introduce Latent On-Policy Self-Distillation (LOPD), which makes the teacher's privileged context itself learnable end-to-end from experience.
  arXiv  Detail & Related papers  (2026-08-13T10:05:51Z)
• Demystifying On-Policy Distillation: Roles, Pathologies, and Regulations [41.8393565183611]
  We show that on-policy distillation (OPD) steers the student toward correct reasoning paths via dense token-level guidance, without expanding capability ceiling.<n>The Student-Teacher Mismatch occurs when a large teacher-student distributional gap causes the guiding signal to misalign with task correctness, steering exploration in counterproductive directions.<n>Length Exploitation arises when the aggregated token-level objective creates length-dependent shortcuts, allowing the student to game the reward landscape through response truncation or redundant padding.<n>To tame these pathologies, we investigate lightweight signal regulations: advantage clipping and log-scale compression,
  arXiv  Detail & Related papers  (2026-07-15T02:52:37Z)
• DemoPSD: Disagreement-Modulated Policy Self-Distillation [49.30809363804017]
  On-policy self-distillation has emerged as a practical method for training large language models.<n>DemoPSD steers the student toward a weighted geometric combination of the teacher and student distributions.
  arXiv  Detail & Related papers  (2026-07-02T17:58:29Z)
• Learning from Own Solutions: Self-Conditioned Credit Assignment for Reinforcement Learning with Verifiable Rewards [49.1203423784326]
  We propose SC- GRPO (Self-Conditioned GRPO), which uses KL divergence mentioned before as a multiplicative weight on GRPO gradients.<n>Across five benchmarks spanning math, code, and agentic tasks, SC- GRPO consistently outperforms 8.1% over GRPO and 5.9% over DAPO with stronger OOD performance.
  arXiv  Detail & Related papers  (2026-06-17T08:26:02Z)
• Localizing Credit at the Divergence: Path-Conditioned Self-Distillation for LLM Reasoning [12.363923974156636]
  Reinforcement learning from verifiable rewards assigns a single scalar to each rollout.<n>On-policy self-distillation addresses this by letting the same model act as a teacher conditioned on privileged information.<n>We propose Hindsight Self-Distillation (HSD), which conditions the teacher on a successful peer rollout.
  arXiv  Detail & Related papers  (2026-06-14T03:37:27Z)
• Agentic Transformers Provably Learn to Search via Reinforcement Learning [80.1412817870784]
  Tree search is a central abstraction behind many language-agent reasoning and decision-making tasks.<n>We study this question in a $k$-ary tree environment, where an agentic transformer observes only its trajectory history through interaction.<n>We first construct a two-head transformer that implements randomized depth-first search (DFS)<n>We then analyze the training dynamics of policy gradient under a depth-wise curriculum, showing that this same DFS mechanism emerges in stages from sparse reinforcement feedback.
  arXiv  Detail & Related papers  (2026-05-29T14:58:03Z)
• StepOPSD: Step-Aware Online Preference Distillation for Agent Reinforcement Learning [6.365332042924078]
  We present StepOPSD, a post-rollout preference self-distillation framework that takes the agent step as the unit of credit redistribution.<n>StepOPSD decomposes trajectories into action-centered step segments, rescoring them under hindsight-enriched teacher contexts.<n>Results reveal a consistent two-knob law: smaller _clip acts as a broadly stabilizing local trust region, whereas the optimal global mixing strength _mix remains task-dependent.
  arXiv  Detail & Related papers  (2026-05-26T15:07:03Z)
• FBOS-RL: Feedback-Driven Bi-Objective Synergistic Reinforcement Learning [16.200486964371713]
  We propose FBOS-RL, a Feedback-Driven Bi-Objective Synergistic reinforcement learning framework.<n>Specifically, we let the model perform Feedback-Guided Exploration Enhancement based on the feedback provided by the environment.<n>Under an identical number of rollouts, FBOS-RL learns substantially faster than GRPO and feedback-based baselines.
  arXiv  Detail & Related papers  (2026-05-18T12:48:36Z)
• Teacher-Guided Policy Optimization for LLM Distillation [90.49982387646861]
  Teacher-Guided Policy Optimization (TGPO) is an on-policy algorithm that incorporates dense directional guidance by leveraging teacher predictions conditioned on the student's rollout.<n>Experiments on complex reasoning benchmarks demonstrate that TGPO significantly outperforms standard baselines and is robust to different teachers.
  arXiv  Detail & Related papers  (2026-05-13T09:20:03Z)
• Truncated Step-Level Sampling with Process Rewards for Retrieval-Augmented Reasoning [32.295907409325615]
  Training large language models to reason with search engines via reinforcement learning is hindered by a credit assignment problem.<n>We propose SLATE, a framework built on two complementary ideas.<n> Experiments on seven QA benchmarks confirm that SLATE consistently outperforms both sparse-reward and process-reward baselines.
  arXiv  Detail & Related papers  (2026-02-26T19:05:40Z)
• Agentic Reinforcement Learning with Implicit Step Rewards [92.26560379363492]
  Large language models (LLMs) are increasingly developed as autonomous agents using reinforcement learning (agentic RL)<n>We introduce implicit step rewards for agentic RL (iStar), a general credit-assignment strategy that integrates seamlessly with standard RL algorithms.<n>We evaluate our method on three challenging agent benchmarks, including WebShop and VisualSokoban, as well as open-ended social interactions with unverifiable rewards in SOTOPIA.
  arXiv  Detail & Related papers  (2025-09-23T16:15:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.