Dynamic Dual-Granularity Skill Bank for Agentic RL

• URL: http://arxiv.org/abs/2603.28716v1
• Date: Mon, 30 Mar 2026 17:32:11 GMT
• Title: Dynamic Dual-Granularity Skill Bank for Agentic RL
• Abstract summary: D2Skill is a dynamic dual-granularity skill bank for agentic reinforcement learning.<n>It organizes reusable experience into task skills for high-level guidance and step skills for fine-grained decision support and error correction.
• Score: 34.161117844675324
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Agentic reinforcement learning (RL) can benefit substantially from reusable experience, yet existing skill-based methods mainly extract trajectory-level guidance and often lack principled mechanisms for maintaining an evolving skill memory. We propose D2Skill, a dynamic dual-granularity skill bank for agentic RL that organizes reusable experience into task skills for high-level guidance and step skills for fine-grained decision support and error correction. D2Skill jointly trains the policy and skill bank through paired baseline and skill-injected rollouts under the same policy, using their performance gap to derive hindsight utility signals for both skill updating and policy optimization. Built entirely from training-time experience, the skill bank is continuously expanded through reflection and maintained with utility-aware retrieval and pruning. Experiments on ALFWorld and WebShop with Qwen2.5-7B-Instruct and Qwen3-4B-Instruct-2507 show that D2Skill consistently improves success rates over skill-free baselines by 10-20 points. Further ablations and analyses show that both dual-granularity skill modeling and dynamic skill maintenance are critical to these gains, while the learned skills exhibit higher utility, transfer across evaluation settings, and introduce only modest training overhead.

Related papers

• SkillForge: Evolving Verifiable Skills for Reinforcement Learning Agents [17.565432600589414]
  Large language model (LLM) agents are trained with reinforcement learning (RL) for complex decision-making tasks.<n>Recent skill-based approaches, such as SkillRL, attempt to address this issue by extracting skills from raw trajectories.<n>We propose SkillForge, a framework for continuous skill evolution that enables skills to be verified and refined through environment interaction.
  arXiv  Detail & Related papers  (2026-08-25T15:52:10Z)
• Bidirectional Context Self-Distillation for Reinforcement Learning of Skill-Based LLM Agents [11.967142343676171]
  We propose BCSD (Bi Context Self-Distillation), a framework that combines self-distillation and reinforcement learning to train agents to use external skills more effectively.<n> Experiments on ALFWorld and WebShop demonstrate that BCSD achieves the strongest overall performance across model scales.
  arXiv  Detail & Related papers  (2026-08-10T12:53:06Z)
• Co-Evolving Skill Generation and Policy Optimization [35.41582114275514]
  Existing methods typically use strong language models to analyze trajectories, generate skills, and update a retrievable skill bank during online training.<n>We propose an online reinforcement learning framework for pre-storage skill validation.
  arXiv  Detail & Related papers  (2026-06-07T17:55:55Z)
• SkillComposer: Learning to Evolve Agent Skills for Specification and Generalization [99.18305770138677]
  Agent skills are reusable strategies that guide agent reasoning and action.<n>Current skill construction methods treat the problem as one-shot extraction.<n>We introduce SkillComposer, a framework that decomposes skill construction into three learnable operations.
  arXiv  Detail & Related papers  (2026-06-04T12:16:08Z)
• Skill is Not One-Size-Fits-All: Model-Aware Skill Alignment for LLM Agents [9.144810798419975]
  LLM agents increasingly retrieve externally curated skills-procedural instructions retrieved at decision time to improve performance on long-horizon interactive tasks.<n>We propose MASA Model-Aware Skill Alignment, a framework that adapts skills to each target backbone without modifying agent weights.<n> MASA consistently achieves the best overall performance, with gains of up to 25.8 points over the strongest baseline.
  arXiv  Detail & Related papers  (2026-05-29T01:34:42Z)
• SKILLC: Learning Autonomous Skill Internalization in LLM Agents via Contrastive Credit Assignment [4.336665585098371]
  We propose SkillC, a framework based on Contrastive Skill Credit Assignment (CSCA) that converts this contrast into a direct learning signal for internalization.<n>textscSkillC samples paired skill-injected and skill-free rollouts for tasks from active skill types within the same policy update.<n>A smoothed validation-level signal further drives an adaptive curriculum over attribution strength, rollout allocation, and monotonic active-set pruning.
  arXiv  Detail & Related papers  (2026-05-27T03:21:19Z)
• Skill-R1: Agent Skill Evolution via Reinforcement Learning [84.35984979949502]
  Skill-R1 is a reinforcement learning framework for instance-level recurrent skill optimization from verifiable rewards.<n>It preserves black-box compatibility with both open- and closed-source models while making adaptation substantially cheaper than model-level updates.
  arXiv  Detail & Related papers  (2026-05-10T06:19:15Z)
• SkillGen: Verified Inference-Time Agent Skill Synthesis [60.927977774369516]
  SkillGen is a framework that synthesizes a single auditable skill from trajectories generated by a base agent.<n>It identifies reusable success patterns, recurring failure modes, and behaviors that appear in nearby successes but are missing from failures.<n>A key novelty in SkillGen is that we model agent skills as interventions to empirically verify the net effect of skills on the overall performance.
  arXiv  Detail & Related papers  (2026-05-09T19:24:11Z)
• SkillOS: Learning Skill Curation for Self-Evolving Agents [67.94374107466957]
  We propose SkillOS, an experience-driven RL training recipe for learning skill curation in self-evolving agents.<n>SkillOS pairs a frozen agent executor that retrieves and applies skills with a trainable skill curator that updates an external SkillRepo from accumulated experience.<n>We show that SkillOS consistently outperforms memory-free and strong memory-based baselines in both effectiveness and efficiency.
  arXiv  Detail & Related papers  (2026-05-07T17:31:50Z)
• XSkill: Continual Learning from Experience and Skills in Multimodal Agents [26.64381741359544]
  XSkill is a dual-stream framework for continual learning from experience and skills in multimodal agents.<n>XSkill grounds both knowledge extraction and retrieval in visual observations.<n>XSkill consistently and substantially outperforms both tool-only and learning-based baselines.
  arXiv  Detail & Related papers  (2026-03-12T15:25:57Z)
• SkillOrchestra: Learning to Route Agents via Skill Transfer [65.50924963973286]
  We introduce SkillOrchestra, a framework for skill-aware orchestration.<n>SkillOrchestra learns fine-grained skills from execution experience and models agent-specific competence and cost under those skills.<n>At deployment, the orchestrator infers the skill demands of the current interaction and selects agents that best satisfy them under an explicit performance-cost trade-off.
  arXiv  Detail & Related papers  (2026-02-23T10:17:25Z)
• SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning [83.98129545309277]
  We propose SkillRL, a framework that bridges the gap between raw experience and policy improvement.<n>Our approach introduces an experience-based distillation mechanism to build a hierarchical skill library SkillBank.<n> Experimental results on ALF, WebShop and seven search-augmented tasks demonstrate that SkillRL achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-02-09T03:17:17Z)
• Skill-Enhanced Reinforcement Learning Acceleration from Heterogeneous Demonstrations [23.628360655654507]
  Learning from Demonstration (LfD) is a well-established problem in Reinforcement Learning (RL)<n>We propose a novel two-stage method dubbed as Skill-enhanced Reinforcement Learning Acceleration (SeRLA)
  arXiv  Detail & Related papers  (2024-12-09T04:58:14Z)
• Skill-Based Reinforcement Learning with Intrinsic Reward Matching [77.34726150561087]
  We present Intrinsic Reward Matching (IRM), which unifies task-agnostic skill pretraining and task-aware finetuning. IRM enables us to utilize pretrained skills far more effectively than previous skill selection methods.
  arXiv  Detail & Related papers  (2022-10-14T00:04:49Z)
• Hierarchical Kickstarting for Skill Transfer in Reinforcement Learning [27.69559938165733]
  Practising and honing skills forms a fundamental component of how humans learn, yet artificial agents are rarely specifically trained to perform them. We investigate how skills can be incorporated into the training of reinforcement learning (RL) agents in complex environments. Our experiments show that learning with a prior knowledge of useful skills can significantly improve the performance of agents on complex problems.
  arXiv  Detail & Related papers  (2022-07-23T19:23:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.