ATOD: Annealed Turn-aware On-policy Distillation for Multi-turn Autonomous Agents
- URL: http://arxiv.org/abs/2606.27814v1
- Date: Fri, 26 Jun 2026 07:56:32 GMT
- Title: ATOD: Annealed Turn-aware On-policy Distillation for Multi-turn Autonomous Agents
- Abstract summary: Training small language-model agents for long-horizon interactive tasks requires both fast imitation and reward-driven improvement.<n>We propose ATOD (Annealed Turn-aware On-policy Distillation), a hybrid online distillation algorithm that explicitly exploits this complementarity.
- Score: 6.4535223115254885
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Training small language-model agents for long-horizon interactive tasks requires both fast imitation and reward-driven improvement. On-policy distillation (OPD) provides dense teacher guidance and typically improves rapidly in the early stage, but its gains saturate once the student approaches the teacher, limiting the final performance ceiling. Reinforcement learning (RL) directly optimizes environment rewards and encourages exploratory improvement toward a higher reward-defined ceiling, but sparse and delayed feedback makes early-stage learning much less efficient than OPD. In this paper, we propose ATOD (Annealed Turn-aware On-policy Distillation), a hybrid online distillation algorithm that explicitly exploits this complementarity. (1) ATOD uses an annealed OPD-RL schedule: OPD dominates early training to approach teacher-level behavior, while RL is gradually strengthened to drive reward-based exploration. (2) ATOD introduces Turn-level Disagreement-Uncertainty Reweighting (T-DUR), which softly amplifies high-utility turns and improves dense supervision in long trajectories. Experiments on ALFWorld, WebShop, and Search-QA show that ATOD consistently outperforms competing post-training baselines: across the three student sizes, ATOD improves average success rate by 3.03 points over OPD and 23.62 points over GRPO, while surpassing the corresponding teacher models by 2.16 points.
Related papers
- TurnOPD: Making On-Policy Distillation Turn-Aware for Efficient Long-Horizon Agent Training [52.6661804966136]
On-policy distillation (OPD) trains a student policy by matching a stronger teacher on the student's own trajectories.<n>We propose TurnOPD, a turn-level budgeting strategy for efficient on-policy distillation of long-horizon agents.
arXiv Detail & Related papers (2026-07-07T03:56:35Z) - On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agents [50.78688247330029]
On-Policy Distillation (OPD) is a natural recipe for transferring such capabilities to smaller students.<n>We propose Guided On-Policy Distillation (Guided-OPD), a simple yet effective algorithm that mixes teacher- and student-generated turns within each rollout.<n>Guided-OPD improves Score by 21.1% and Success Rate by 25.5% over vanilla OPD on average.
arXiv Detail & Related papers (2026-06-14T16:41:45Z) - Are Full Rollouts Necessary for On-Policy Distillation? [63.18243901591995]
On-policy distillation (OPD) provides dense teacher feedback along student-generated rollouts.<n>We identify the rollout horizon as a key bottleneck in OPD that substantially impacts training efficiency.<n>We propose two simple horizon-control strategies: Progressive OPD, which gradually expands the rollout horizon during training, and Truncated OPD, which permanently performs distillation on reliable truncated rollouts.
arXiv Detail & Related papers (2026-05-29T16:12:54Z) - Self-Distilled Agentic Reinforcement Learning [65.24201057390938]
Self-Distilled Agentic Reinforcement Learning maps detached token-level signals into a sigmoid gate.<n>SDAR consistently outperforms hybrid RL--OPSD baselines across model scales.
arXiv Detail & Related papers (2026-05-14T17:51:26Z) - Learning to Foresee: Unveiling the Unlocking Efficiency of On-Policy Distillation [51.210887267509854]
We argue that OPD's efficiency stems from a form of foresight'': it establishes a stable update trajectory toward the final model early in training.<n>We propose textbfEffOPD, a plug-and-play acceleration method that speeds up OPD by adaptively selecting an extrapolation step size and moving along the current update direction.
arXiv Detail & Related papers (2026-05-12T08:19:15Z) - SOD: Step-wise On-policy Distillation for Small Language Model Agents [32.49707795291693]
Tool-integrated reasoning is difficult to scale to small language models.<n>Recently, on-policy distillation has gained popularity by supplying dense token-level supervision from a teacher.<n>We propose SOD, a step-wise on-policy distillation framework for small language model agents.
arXiv Detail & Related papers (2026-05-08T13:30:42Z) - Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation [57.524909883706556]
On-policy distillation (OPD) has demonstrated strong empirical gains in improving student performance.<n>This work introduces a flexible reference model and a reward scaling factor that controls the relative weight of the reward term against the KL regularization.<n>In particular, in the setting where we merge the knowledge from different domain experts, obtained by applying domain-specific RL to the same student model, ExOPD enables the student to even surpass the teacher's performance boundary.
arXiv Detail & Related papers (2026-02-12T16:14:29Z) - Video-OPD: Efficient Post-Training of Multimodal Large Language Models for Temporal Video Grounding via On-Policy Distillation [29.755136665244805]
Reinforcement learning has emerged as a principled post-training paradigm for Temporal Video Grounding (TVG)<n>We propose Video-OPD, an efficient post-training framework for TVG inspired by recent advances in on-policy distillation.
arXiv Detail & Related papers (2026-02-03T02:05:48Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.