Revisiting DAgger in the Era of LLM-Agents

• URL: http://arxiv.org/abs/2605.12913v1
• Date: Wed, 13 May 2026 02:40:28 GMT
• Title: Revisiting DAgger in the Era of LLM-Agents
• Abstract summary: Long-horizon LM agents learn from multi-turn interaction, where a single early mistake can alter the subsequent state distribution and derail the whole trajectory.<n>Existing recipes fall short in complementary ways: supervised fine-tuning provides dense teacher supervision, while reinforcement learning with verifiable rewards avoids this off-policy mismatch.<n>We address this dilemma by revisiting the dataset aggregation (DAgger) for multi-turn LM agents.
• Score: 35.615579397673166
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Long-horizon LM agents learn from multi-turn interaction, where a single early mistake can alter the subsequent state distribution and derail the whole trajectory. Existing recipes fall short in complementary ways: supervised fine-tuning provides dense teacher supervision but suffers from covariate shift because it is trained on off-policy teacher trajectories; while reinforcement learning with verifiable rewards avoids this off-policy mismatch by learning from on-policy rollouts but with only sparse outcome feedback. We address this dilemma by revisiting Dataset Aggregation (DAgger) for multi-turn LM agents: the algorithm collects trajectories through a turn-level interpolation of student and teacher policies, and the student is then trained on these trajectories using supervised labels provided by the teacher. By directly interacting with environments, we expose the model to realistic states likely to be encountered during deployment, thereby effectively mitigating covariate shift. Besides, since the student is learned by mimicking the teacher's behavior, it receives rich feedback during learning. To demonstrate DAgger enjoys the benefits of both worlds, we tested the algorithm to train a software-engineering agent with 4B- and 8B-scale student models. On SWE-bench Verified, our DAgger-style training improves over the strongest post-training baseline by +3.9 points at 4B and +3.6 points at 8B. The resulting 4B agent reaches 27.3%, outperforming representative published 8B SWE-agent systems, while the 8B agent achieves 29.8%, surpassing SWE-Gym-32B and coming within 5 points of stronger 32B-scale agents. Together with consistent gains on the held-out SWE-Gym split, these results suggest the effectiveness of DAgger for modern long-horizon LM agents.

Related papers

• From Scoring to Acting: Outcome-Verified Comparative Self-Distillation for LLM Agents [16.67357658768852]
  We propose Outcome-Verified Comparative Self-Distillation (OVCSD)<n>OVCSD organizes failed student rollouts into a prefix tree, adaptively invokes a skill-conditioned teacher from student-reached states, and retains only outcome-verified successful continuations.<n> Experiments on ALFWorld and WebShop across three model scales show that OVCSD consistently outperforms skill-free RL and existing self-distillation baselines.
  arXiv  Detail & Related papers  (2026-07-30T09:47:58Z)
• A Learning-Rate-Gated Failure of GRPO in a Small Language and Vision-Language Model Web Agent: A Controlled Null and Its Mechanism [11.543353599329675]
  Reinforcement learning with verifiable rewards is now run routinely on a supervised checkpoint in the hope of producing a stronger agent.<n>We ask whether it adds skill to a small language and vision-language model web agent at the 4B to 8B scale.<n>No configuration credibly improves the success rate of a strong supervised baseline on tasks the agent has largely mastered.
  arXiv  Detail & Related papers  (2026-07-14T11:17:35Z)
• On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agents [50.78688247330029]
  On-Policy Distillation (OPD) is a natural recipe for transferring such capabilities to smaller students.<n>We propose Guided On-Policy Distillation (Guided-OPD), a simple yet effective algorithm that mixes teacher- and student-generated turns within each rollout.<n>Guided-OPD improves Score by 21.1% and Success Rate by 25.5% over vanilla OPD on average.
  arXiv  Detail & Related papers  (2026-06-14T16:41:45Z)
• What Makes Interaction Trajectories Effective for Training Terminal Agents? [55.62817294510983]
  Terminal-Lego is a scalable pipeline that transforms real-world issues into environment-verified agentic tasks.<n>Students fine-tuned on trajectories from DeepSeek-V3.2, a lower-scoring agent, exhibit significantly stronger generalization.<n>With only 15.3k Terminal-Lego trajectories, Qwen3-32B achieves a 24.3% score on Terminal-Bench 2.0, rivaling previous SOTA performance established with over 30x the data volume.
  arXiv  Detail & Related papers  (2026-06-02T10:37:47Z)
• Self-Distilled Agentic Reinforcement Learning [65.24201057390938]
  Self-Distilled Agentic Reinforcement Learning maps detached token-level signals into a sigmoid gate.<n>SDAR consistently outperforms hybrid RL--OPSD baselines across model scales.
  arXiv  Detail & Related papers  (2026-05-14T17:51:26Z)
• MAD-OPD: Breaking the Ceiling in On-Policy Distillation via Multi-Agent Debate [17.31967380175203]
  We propose MAD-OPD (Multi-Agent Debate-driven On-Policy Distillation), which recasts the distillation teacher as a deliberative collective of teachers that debate over the student's on-policy state.<n>We also introduce On-Policy Agentic Distillation (OPAD), which adds step-level sampling to stabilize training under multi-step error compounding.
  arXiv  Detail & Related papers  (2026-05-02T09:41:37Z)
• TCOD: Exploring Temporal Curriculum in On-Policy Distillation for Multi-turn Autonomous Agents [55.27396165691312]
  We identify a key limitation of vanilla OPD in multi-turn agent settings, which we term Trajectory-Level KL Instability.<n>We propose TCOD, a framework that controls the trajectory depth exposed to the student and progressively expands it with a curriculum schedule.<n> Experimental results across four student-teacher pairs show that TCOD mitigates KL escalation and enhances KL stability throughout training, improving agent performance by up to 18 points over vanilla OPD.
  arXiv  Detail & Related papers  (2026-04-27T03:38:27Z)
• SCOPE: Signal-Calibrated On-Policy Distillation Enhancement with Dual-Path Adaptive Weighting [17.504616835765617]
  We propose a dual-path adaptive training framework that routes on-policy rollouts by correctness into two complementary supervision paths.<n>SCOPE achieves an average relative improvement of 11.42% in Avg@32 and 7.30% in Pass@32 over competitive baselines.
  arXiv  Detail & Related papers  (2026-04-12T15:26:14Z)
• A Rubric-Supervised Critic from Sparse Real-World Outcomes [87.11204512676193]
  Real-world coding agents operate with humans in the loop, where success signals are typically noisy, delayed, and sparse.<n>We propose a process to learn a "critic" model from sparse and noisy interaction data, which can then be used both as a reward model for either RL-based training or inference-time scaling.
  arXiv  Detail & Related papers  (2026-03-04T07:23:54Z)
• Agentic Reinforcement Learning with Implicit Step Rewards [92.26560379363492]
  Large language models (LLMs) are increasingly developed as autonomous agents using reinforcement learning (agentic RL)<n>We introduce implicit step rewards for agentic RL (iStar), a general credit-assignment strategy that integrates seamlessly with standard RL algorithms.<n>We evaluate our method on three challenging agent benchmarks, including WebShop and VisualSokoban, as well as open-ended social interactions with unverifiable rewards in SOTOPIA.
  arXiv  Detail & Related papers  (2025-09-23T16:15:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.