Bidirectional Context Self-Distillation for Reinforcement Learning of Skill-Based LLM Agents

• URL: http://arxiv.org/abs/2608.09555v1
• Date: Mon, 10 Aug 2026 12:53:06 GMT
• Title: Bidirectional Context Self-Distillation for Reinforcement Learning of Skill-Based LLM Agents
• Abstract summary: We propose BCSD (Bi Context Self-Distillation), a framework that combines self-distillation and reinforcement learning to train agents to use external skills more effectively.<n> Experiments on ALFWorld and WebShop demonstrate that BCSD achieves the strongest overall performance across model scales.
• Score: 11.967142343676171
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: External natural-language skills provide large language model (LLM) agents with reusable and editable guidance for solving complex tasks. Yet their effectiveness depends not only on skill quality, but also on whether the policy can translate the provided guidance into appropriate actions. However, methods specifically designed to improve this skill-utilization ability remain largely underexplored. In practice, skill-based agents are commonly trained with reinforcement learning objectives centered on task-level rewards, which offer limited supervision and struggle to capture subtle differences in how effectively the policy uses the provided skills. We propose BCSD (Bidirectional Context Self-Distillation), a framework that combines self-distillation with reinforcement learning to train LLM agents to use external skills more effectively. Unlike prior self-distillation methods that rely on a single privileged context, BCSD evaluates each trajectory from two complementary skill-context views. The augmented view introduces higher-level Meta-Skill guidance, while the reduced view prunes general guidance to highlight task-specific skills. Their complementary token-level signals are combined to rescale the RL advantage. Experiments on ALFWorld and WebShop demonstrate that BCSD achieves the strongest overall performance across model scales, enabling agents to utilize external skills more effectively. Ablation studies further verify the complementary contributions of the augmented and reduced context views. Code will be released to ensure full reproducibility.

Related papers

• AUSO: Action-Level Unified Skill Optimization from Internalization to Utilization [23.057686255420546]
  We introduce AUSO, which skill learning and skill use through a progressive, action-aware optimization process.<n>At the beginning of training, AUSO jointly learns from teacher guidance and environmental outcomes.<n>As the policy matures, AUSO evaluates each sampled action under both skill-conditioned and skill-free contexts.
  arXiv  Detail & Related papers  (2026-08-21T16:53:27Z)
• Evo-Harness: Context-to-Harness Skill Compilation for Self-Evolving Agents [67.97365535829098]
  Learning from experience is critical for developing capable, self-improving large language model (LLM) agents.<n>We propose Evo-Harness, which distills noisy, single-shot executions into reusable skill harnesses for cross-domain and topic-level adaptation.<n>Our analysis demonstrates the effectiveness of Evo-Harness and provides a principled understanding of how LLM agents can effectively learn on the fly.
  arXiv  Detail & Related papers  (2026-08-15T06:43:56Z)
• Latent On-Policy Self-Distillation [55.57800223145407]
  On-policy self-distillation (OPSD) offers an effective pathway by using a privileged self-teacher to provide dense supervision on the student's own trajectories.<n>Existing methods still rely heavily on designer-specified privileged artifacts.<n>We introduce Latent On-Policy Self-Distillation (LOPD), which makes the teacher's privileged context itself learnable end-to-end from experience.
  arXiv  Detail & Related papers  (2026-08-13T10:05:51Z)
• OPID: On-Policy Skill Distillation for Agentic Reinforcement Learning [38.54413500261524]
  On-policy self-distillation offers dense token-level supervision.<n>textbfOPID (textbfOn-textbfPolicy Sktextbfill textbfDistillation) is a framework that extracts skill supervision directly from completed on trajectories.
  arXiv  Detail & Related papers  (2026-06-25T09:24:09Z)
• Skill0.5: Joint Skill Internalization and Utilization for Out-of-Distribution Generalization in Agentic Reinforcement Learning [25.966432263031397]
  Skill0.5 is a novel agentic RL framework that explicitly differentiates skill treatments by combining general skill internalization with task-specific skill utilization.<n>It internalizes general skills via privileged distillation to build a cognitive foundation for hard tasks, while using diagnostic probing on easy tasks to penalize shortcuts and enforce specific skill utilization.<n> Experiments on ALFWorld and WebShop demonstrate that Skill0.5 outperforms both memory-based and skill-based RL baselines.
  arXiv  Detail & Related papers  (2026-05-27T12:54:33Z)
• SKILLC: Learning Autonomous Skill Internalization in LLM Agents via Contrastive Credit Assignment [4.336665585098371]
  We propose SkillC, a framework based on Contrastive Skill Credit Assignment (CSCA) that converts this contrast into a direct learning signal for internalization.<n>textscSkillC samples paired skill-injected and skill-free rollouts for tasks from active skill types within the same policy update.<n>A smoothed validation-level signal further drives an adaptive curriculum over attribution strength, rollout allocation, and monotonic active-set pruning.
  arXiv  Detail & Related papers  (2026-05-27T03:21:19Z)
• Dynamic Skill Lifecycle Management for Agentic Reinforcement Learning [11.76119198695392]
  Large language model agents increasingly rely on external skills to solve complex tasks.<n>Existing methods assume external skills either accumulate as persistent guidance or internalized into the policy, leading to zero-skill inference.<n>We argue this assumption is overly restrictive, since with limited parametric capacity and uneven marginal contribution across skills, the optimal active skill set is non-monotonic, task- and stage-dependent.
  arXiv  Detail & Related papers  (2026-05-11T17:55:13Z)
• Skill-R1: Agent Skill Evolution via Reinforcement Learning [84.35984979949502]
  Skill-R1 is a reinforcement learning framework for instance-level recurrent skill optimization from verifiable rewards.<n>It preserves black-box compatibility with both open- and closed-source models while making adaptation substantially cheaper than model-level updates.
  arXiv  Detail & Related papers  (2026-05-10T06:19:15Z)
• From Context to Skills: Can Language Models Learn from Context Skillfully? [56.5857437218136]
  Ctx2Skill is a self-evolving framework that autonomously discovers, refines, and selects context-specific skills.<n>It can be plugged into any language model to obtain better context learning capability.
  arXiv  Detail & Related papers  (2026-04-30T09:53:15Z)
• SKILL0: In-Context Agentic Reinforcement Learning for Skill Internalization [61.37478254738943]
  Inference-time skill augmentation is fundamentally limited.<n>We introduce SKILL0, an in-context reinforcement learning framework designed for skill internalization.<n>SKILL0 achieves substantial improvements over the standard RL baseline.
  arXiv  Detail & Related papers  (2026-04-02T17:03:05Z)
• Acquiring Diverse Skills using Curriculum Reinforcement Learning with Mixture of Experts [58.220879689376744]
  Reinforcement learning (RL) is a powerful approach for acquiring a good-performing policy. We propose textbfDiverse textbfSkill textbfLearning (Di-SkilL) for learning diverse skills. We show on challenging robot simulation tasks that Di-SkilL can learn diverse and performant skills.
  arXiv  Detail & Related papers  (2024-03-11T17:49:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.