AHEAD: Adaptive Hindsight with Environment-Augmented Distillation for Agentic RL

• URL: http://arxiv.org/abs/2608.24114v1
• Date: Tue, 25 Aug 2026 06:23:48 GMT
• Title: AHEAD: Adaptive Hindsight with Environment-Augmented Distillation for Agentic RL
• Abstract summary: Self-distillation methods can provide finer-grained supervision by augmenting RL with privileged information.<n>We propose AHEAD, a step-aware framework that matches different supervision sources to different step types.<n>AHEAD raises task success (+13.3 points on ALFWorld), reaches a given success rate in fewer training steps, and solves tasks within tighter interaction budgets than outcome-only RL and prior self-distillation baselines.
• Score: 14.402945966056764
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Training multi-turn LLM agents with reinforcement learning typically relies on trajectory-level rewards, which assign a uniform advantage to every step and cannot identify which decisions led to success or failure. Self-distillation methods can provide finer-grained supervision by augmenting RL with privileged information. However, existing approaches usually apply the same type of privileged information to every step in an indistinguishable manner, ignoring a key asymmetry: routine steps need little additional guidance, while critical error steps require corrective direction that environment feedback alone cannot provide. We propose AHEAD, a step-aware framework that matches different supervision sources to different step types. The teacher receives environment feedback on all steps as a grounded dense signal, and additionally receives LLM-generated corrective hints on error steps to supply the direction that environment feedback lacks. The method introduces minimal changes to the standard GRPO algorithm. Across ALFWorld, WebShop, and Search-based QA, and across three model scales, AHEAD raises task success (+13.3 points on ALFWorld and +11.0 on WebShop at 7B over GRPO), reaches a given success rate in fewer training steps, and solves tasks within tighter interaction budgets than outcome-only RL and prior self-distillation baselines.

Related papers

• Beyond Outcome Rewards: Step-Level Self-Distilled Policy Optimization for Deep Search Agents [15.62688304085298]
  Deep search agents operate over trajectories spanning dozens of steps.<n>On-policy self-distillation (OPSD) addresses this by using the model's own logits as dense token-level teachers.<n>We propose Step-Level Self-Distilled Policy Optimization (SSPO), which converts teacher-student disagreement into step-level advantage weights.
  arXiv  Detail & Related papers  (2026-08-13T03:13:18Z)
• LLM-as-a-Tutor: Policy-Aware Prompt Adaptation for Non-Verifiable RL [64.4276583943816]
  Reinforcement learning for non-verifiable instruction relies on judges with prompt-specific rubrics as reward signals.<n>While recent methods adapt these rubrics to the evolving policy during training, the training prompts themselves remain static.<n>This static approach often results in a critical misalignment between prompt difficulty and policy capability, leaving the judge unable to recover a discriminative reward signal.
  arXiv  Detail & Related papers  (2026-07-05T17:05:13Z)
• Self-evolving LLM agents with in-distribution Optimization [60.05867547965365]
  Large Language Models (LLMs) have recently emerged as powerful controllers for interactive agents in complex environments.<n>We propose Q-Evolve, a self-evolving framework for LLM agents that unifies automatic process-reward labeling and policy learning.<n>We evaluate our method on AlfWorld, WebShop, and ScienceWorld, showing Q-Evolve outperforms strong baselines in sample efficiency, robustness, and overall task performance.
  arXiv  Detail & Related papers  (2026-06-05T15:09:52Z)
• What and When to Distill: Selective Hindsight Distillation for Multi-Turn Agents [70.6980022118038]
  Reinforcement learning can train LLM agents from sparse task rewards, but long-horizon credit assignment remains challenging.<n>Existing methods rely on trajectory-level rewards or proxy signals, without fully leveraging per-step environmental feedback.<n>We introduce SERL, a selective environment-reweighted learning framework.
  arXiv  Detail & Related papers  (2026-05-19T07:00:55Z)
• GradAlign: Gradient-Aligned Data Selection for LLM Reinforcement Learning [55.03441672267886]
  We propose GradAlign, a gradient-aligned data selection method for reinforcement learning.<n>We evaluate GradAlign across three data regimes: unreliable reward signals, distribution imbalance, and low-utility training corpus.
  arXiv  Detail & Related papers  (2026-02-25T01:54:50Z)
• Verified Critical Step Optimization for LLM Agents [67.05296684575445]
  Critical Step Optimization focuses preference learning on verified critical steps.<n>Method starts from failed policy trajectories rather than expert demonstrations.<n>Experiments on GAIA-Text-103 and XBench-DeepSearch show that CSO achieves 37% and 26% relative improvement over the SFT baseline.
  arXiv  Detail & Related papers  (2026-02-03T11:41:02Z)
• ProRAG: Process-Supervised Reinforcement Learning for Retrieval-Augmented Generation [54.071574153853994]
  ProRAG is a process-supervised reinforcement learning framework designed to integrate learned step-level supervision into the online optimization loop.<n>Our framework consists of four stages: (1) Supervised Policy Warmup to initialize the model with a structured reasoning format; (2) construction of an MCTS-based Process Reward Model (PRM) to quantify intermediate reasoning quality; (3) PRM-Guided Reasoning Refinement to align the policy with fine-grained process preferences; and (4) Process-Supervised Reinforcement Learning with a dual-granularity advantage mechanism.
  arXiv  Detail & Related papers  (2026-01-29T16:04:59Z)
• SALT: Step-level Advantage Assignment for Long-horizon Agents via Trajectory Graph [15.303565124424635]
  Large Language Models (LLMs) have demonstrated remarkable capabilities, enabling language agents to excel at single-turn tasks.<n>Their application to complex, multi-step, and long-horizon tasks remains challenging.<n>We propose SALT, a framework that provides a finer-grained advantage assignment, derived solely from outcome rewards.
  arXiv  Detail & Related papers  (2025-10-22T20:50:24Z)
• ATLaS: Agent Tuning via Learning Critical Steps [39.279048406057264]
  Large Language Model (LLM) agents have demonstrated remarkable generalization capabilities across multi-domain tasks.<n>Existing agent tuning approaches typically employ supervised finetuning on entire expert trajectories.<n>We propose ATLaS that identifies the critical steps in expert trajectories and finetunes LLMs solely on these steps with reduced costs.
  arXiv  Detail & Related papers  (2025-03-04T02:14:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.