Co-Evolving Skill Generation and Policy Optimization

• URL: http://arxiv.org/abs/2606.08755v1
• Date: Sun, 07 Jun 2026 17:55:55 GMT
• Title: Co-Evolving Skill Generation and Policy Optimization
• Abstract summary: Existing methods typically use strong language models to analyze trajectories, generate skills, and update a retrievable skill bank during online training.<n>We propose an online reinforcement learning framework for pre-storage skill validation.
• Score: 35.41582114275514
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Skill-augmented reinforcement learning improves language agents by storing reusable procedural knowledge acquired from past experience. Existing methods typically use strong language models to analyze trajectories, generate skills, and update a retrievable skill bank during online training. However, they rarely assess whether a newly generated skill is useful before it is stored and reused. We find that this assumption is unreliable: even skills generated by proprietary frontier LLMs exhibit highly mixed utility, with many providing little benefit or even degrading performance. Once such skills enter the bank, their effects are difficult to identify, because subsequent rollout feedback is delayed and usually reflects the combined effect of multiple retrieved skills rather than the marginal contribution of any individual skill. We propose an online reinforcement learning framework for pre-storage skill validation. The framework estimates whether a candidate skill contributes useful information beyond the skills already retrieved for the current task. It uses the standard rollout budget to form two matched groups under the same task and retrieval context: base rollouts conditioned on the currently retrieved skills, and skill-augmented rollouts conditioned on the same skills plus one candidate skill induced from the base trajectories. The reward gap between these two groups estimates the candidate skill's context-dependent marginal utility, enabling the framework to promote useful skills while filtering ineffective or harmful ones without additional rollout overhead. The framework further uses this marginal-utility signal to train the policy itself as a skill generator, reducing reliance on repeated calls to proprietary models. The learned skill-generation likelihood serves as a context-dependent score for retrieval-time reranking and outdated-skill pruning as the policy evolves.

Related papers

• SkillForge: Evolving Verifiable Skills for Reinforcement Learning Agents [17.565432600589414]
  Large language model (LLM) agents are trained with reinforcement learning (RL) for complex decision-making tasks.<n>Recent skill-based approaches, such as SkillRL, attempt to address this issue by extracting skills from raw trajectories.<n>We propose SkillForge, a framework for continuous skill evolution that enables skills to be verified and refined through environment interaction.
  arXiv  Detail & Related papers  (2026-08-25T15:52:10Z)
• SkillAligner: Treating Retrieved Skills as Adaptable Drafts at Execution Time [14.34844805305673]
  General-purpose skills promise reusable procedural knowledge for language agents.<n>But semantic relevance does not guarantee execution utility.<n>We propose SkillAligner, a training-free execution-time skill adaptation framework.
  arXiv  Detail & Related papers  (2026-08-07T07:05:53Z)
• SKILLC: Learning Autonomous Skill Internalization in LLM Agents via Contrastive Credit Assignment [4.336665585098371]
  We propose SkillC, a framework based on Contrastive Skill Credit Assignment (CSCA) that converts this contrast into a direct learning signal for internalization.<n>textscSkillC samples paired skill-injected and skill-free rollouts for tasks from active skill types within the same policy update.<n>A smoothed validation-level signal further drives an adaptive curriculum over attribution strength, rollout allocation, and monotonic active-set pruning.
  arXiv  Detail & Related papers  (2026-05-27T03:21:19Z)
• Skill-R1: Agent Skill Evolution via Reinforcement Learning [84.35984979949502]
  Skill-R1 is a reinforcement learning framework for instance-level recurrent skill optimization from verifiable rewards.<n>It preserves black-box compatibility with both open- and closed-source models while making adaptation substantially cheaper than model-level updates.
  arXiv  Detail & Related papers  (2026-05-10T06:19:15Z)
• SkillGen: Verified Inference-Time Agent Skill Synthesis [60.927977774369516]
  SkillGen is a framework that synthesizes a single auditable skill from trajectories generated by a base agent.<n>It identifies reusable success patterns, recurring failure modes, and behaviors that appear in nearby successes but are missing from failures.<n>A key novelty in SkillGen is that we model agent skills as interventions to empirically verify the net effect of skills on the overall performance.
  arXiv  Detail & Related papers  (2026-05-09T19:24:11Z)
• SkillLens: Adaptive Multi-Granularity Skill Reuse for Cost-Efficient LLM Agents [6.293920920196533]
  SkillLens is a hierarchical skill-evolution framework that organizes skills into a four-layer graph of policies, strategies, procedures, and primitives.<n>It retrieves semantically relevant skill seeds, expands them through degree-corrected random walk over the skill graph, and then uses a verifier to decide whether each visited unit should be accepted, decomposed, rewritten, or skipped.<n>Across MuLocbench and ALFWorld, SkillLens consistently improves over strong skill-based baselines.
  arXiv  Detail & Related papers  (2026-05-08T18:48:04Z)
• SkillOS: Learning Skill Curation for Self-Evolving Agents [67.94374107466957]
  We propose SkillOS, an experience-driven RL training recipe for learning skill curation in self-evolving agents.<n>SkillOS pairs a frozen agent executor that retrieves and applies skills with a trainable skill curator that updates an external SkillRepo from accumulated experience.<n>We show that SkillOS consistently outperforms memory-free and strong memory-based baselines in both effectiveness and efficiency.
  arXiv  Detail & Related papers  (2026-05-07T17:31:50Z)
• From Context to Skills: Can Language Models Learn from Context Skillfully? [56.5857437218136]
  Ctx2Skill is a self-evolving framework that autonomously discovers, refines, and selects context-specific skills.<n>It can be plugged into any language model to obtain better context learning capability.
  arXiv  Detail & Related papers  (2026-04-30T09:53:15Z)
• SKILL0: In-Context Agentic Reinforcement Learning for Skill Internalization [61.37478254738943]
  Inference-time skill augmentation is fundamentally limited.<n>We introduce SKILL0, an in-context reinforcement learning framework designed for skill internalization.<n>SKILL0 achieves substantial improvements over the standard RL baseline.
  arXiv  Detail & Related papers  (2026-04-02T17:03:05Z)
• Skill-Based Reinforcement Learning with Intrinsic Reward Matching [77.34726150561087]
  We present Intrinsic Reward Matching (IRM), which unifies task-agnostic skill pretraining and task-aware finetuning. IRM enables us to utilize pretrained skills far more effectively than previous skill selection methods.
  arXiv  Detail & Related papers  (2022-10-14T00:04:49Z)
• Hierarchical Kickstarting for Skill Transfer in Reinforcement Learning [27.69559938165733]
  Practising and honing skills forms a fundamental component of how humans learn, yet artificial agents are rarely specifically trained to perform them. We investigate how skills can be incorporated into the training of reinforcement learning (RL) agents in complex environments. Our experiments show that learning with a prior knowledge of useful skills can significantly improve the performance of agents on complex problems.
  arXiv  Detail & Related papers  (2022-07-23T19:23:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.