Skill Set Optimization: Reinforcing Language Model Behavior via Transferable Skills

• URL: http://arxiv.org/abs/2402.03244v2
• Date: Sat, 22 Jun 2024 18:58:09 GMT
• Title: Skill Set Optimization: Reinforcing Language Model Behavior via Transferable Skills
• Abstract summary: Large language models (LLMs) have recently been used for sequential decision making in interactive environments. We propose Skill Set Optimization (SSO) for improving LLM actor performance through constructing and refining sets of transferable skills. We evaluate our method in the classic videogame NetHack and the text environment ScienceWorld to demonstrate SSO's ability to optimize a set of skills and perform in-context policy improvement.
• Score: 40.823689847227975
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) have recently been used for sequential decision making in interactive environments. However, leveraging environment reward signals for continual LLM actor improvement is not straightforward. We propose Skill Set Optimization (SSO) for improving LLM actor performance through constructing and refining sets of transferable skills. SSO constructs skills by extracting common subtrajectories with high rewards and generating subgoals and instructions to represent each skill. These skills are provided to the LLM actor in-context to reinforce behaviors with high rewards. Then, SSO further refines the skill set by pruning skills that do not continue to result in high rewards. We evaluate our method in the classic videogame NetHack and the text environment ScienceWorld to demonstrate SSO's ability to optimize a set of skills and perform in-context policy improvement. SSO outperforms baselines by 40% in our custom NetHack task and outperforms the previous state-of-the-art in ScienceWorld by 35%.

Related papers

• SkillsInjector: Dynamic Skill Context Construction for LLM Agents [16.631471381875816]
  Existing methods treat skill injection as a static step, selecting skills with fixed criteria, fixing the budget in advance, and leaving descriptions unchanged.<n>We propose SkillsInjector, a two-stage adaptive method that jointly addresses these decisions.<n>Across tau2-bench, SkillsBench, and ALFWorld, SkillsInjector achieves the highest score, improving over the strongest baseline by 3.9, 6.1, and 7.3 percentage points, respectively.
  arXiv  Detail & Related papers  (2026-05-28T11:44:32Z)
• Skill-R1: Agent Skill Evolution via Reinforcement Learning [84.35984979949502]
  Skill-R1 is a reinforcement learning framework for instance-level recurrent skill optimization from verifiable rewards.<n>It preserves black-box compatibility with both open- and closed-source models while making adaptation substantially cheaper than model-level updates.
  arXiv  Detail & Related papers  (2026-05-10T06:19:15Z)
• Co-Evolving LLM Decision and Skill Bank Agents for Long-Horizon Tasks [83.4942519570046]
  Large Language Models (LLMs) offer a promising alternative as game playing agents, but they often struggle with consistent long horizon decision making.<n>We present COSPLAY, a co evolution framework in which an LLM decision agent retrieves skills from a learnable skill bank to guide action taking.<n>Our framework improves both the decision agent to learn better skill retrieval and action generation, while the skill bank agent continually extracts, refines, and updates skills together with their contracts.
  arXiv  Detail & Related papers  (2026-04-22T18:17:17Z)
• GraSP: Graph-Structured Skill Compositions for LLM Agents [29.818044584102456]
  Skill ecosystems for LLM agents have matured rapidly, yet recent benchmarks show that providing agents with more skills does not monotonically improve performance.<n>We propose GraSP, the first executable skill graph architecture that introduces a compilation layer between skill retrieval and execution.<n>GraSP transforms flat skill sets into typed directed acyclic graphs (DAGs) with precondition-effect edges, executes them with node-level verification, and performs locality-bounded repair through five typed operators.
  arXiv  Detail & Related papers  (2026-04-20T06:31:11Z)
• How Well Do Agentic Skills Work in the Wild: Benchmarking LLM Skill Usage in Realistic Settings [29.3744517703302]
  We study the first comprehensive study of skill utility under progressively challenging realistic settings.<n>Our findings reveal that performance gains degrade consistently as settings become more realistic.<n>We show that query-specific refinement substantially recovers lost performance when the initial skills are of reasonable relevance and quality.
  arXiv  Detail & Related papers  (2026-04-06T00:10:30Z)
• SKILL0: In-Context Agentic Reinforcement Learning for Skill Internalization [61.37478254738943]
  Inference-time skill augmentation is fundamentally limited.<n>We introduce SKILL0, an in-context reinforcement learning framework designed for skill internalization.<n>SKILL0 achieves substantial improvements over the standard RL baseline.
  arXiv  Detail & Related papers  (2026-04-02T17:03:05Z)
• GIFT: Games as Informal Training for Generalizable LLMs [64.47890325824763]
  Large Language Models (LLMs) struggle with "practical wisdom" and generalizable intelligence.<n>This gap arises from a lack of informal learning, which thrives on interactive feedback rather than goal-oriented instruction.<n>We propose treating Games as a primary environment for LLM informal learning, leveraging their intrinsic reward signals and abstracted complexity.
  arXiv  Detail & Related papers  (2026-01-09T08:42:44Z)
• Reinforcement Learning for Self-Improving Agent with Skill Library [14.717149089634718]
  Large Language Model (LLM)-based agents have demonstrated remarkable capabilities in complex reasoning and multi-turn interactions.<n>One promising approach is implementing skill libraries that allow agents to learn, validate, and apply new skills.<n>We propose a Reinforcement Learning (RL)-based approach to enhance agents' self-improvement capabilities with a skill library.
  arXiv  Detail & Related papers  (2025-12-18T21:58:19Z)
• Learning Adaptive Dexterous Grasping from Single Demonstrations [27.806856958659054]
  This work tackles two key challenges: efficient skill acquisition from limited human demonstrations and context-driven skill selection. AdaDexGrasp learns a library of grasping skills from a single human demonstration per skill and selects the most suitable one using a vision-language model (VLM) We evaluate AdaDexGrasp in both simulation and real-world settings, showing that our approach significantly improves RL efficiency and enables learning human-like grasp strategies across varied object configurations.
  arXiv  Detail & Related papers  (2025-03-26T04:05:50Z)
• LLM Post-Training: A Deep Dive into Reasoning Large Language Models [131.10969986056]
  Large Language Models (LLMs) have transformed the natural language processing landscape and brought to life diverse applications. Post-training methods enable LLMs to refine their knowledge, improve reasoning, enhance factual accuracy, and align more effectively with user intents and ethical considerations.
  arXiv  Detail & Related papers  (2025-02-28T18:59:54Z)
• Self-Improving Skill Learning for Robust Skill-based Meta-Reinforcement Learning [6.959686714606018]
  Self-Improving Skill Learning (SISL) performs self-guided skill refinement using decoupled high-level and skill improvement policies.<n>SISL consistently outperforms other skill-based meta-RL methods on diverse long-horizon tasks.
  arXiv  Detail & Related papers  (2025-02-06T03:28:45Z)
• MaestroMotif: Skill Design from Artificial Intelligence Feedback [67.17724089381056]
  MaestroMotif is a method for AI-assisted skill design, which yields high-performing and adaptable agents. We present MaestroMotif, a method for AI-assisted skill design, which yields high-performing and adaptable agents.
  arXiv  Detail & Related papers  (2024-12-11T16:59:31Z)
• LLM2CLIP: Powerful Language Model Unlocks Richer Visual Representation [72.02635550088546]
  This work explores how large language models (LLMs) can enhance CLIP's capability, especially for processing longer and more complex image captions.<n>We introduce a caption-to-caption contrastive fine-tuning framework, significantly enhancing the discriminative quality of LLM outputs.<n>Our approach outperforms LoRA-based methods, achieving nearly fourfold faster training with superior performance.
  arXiv  Detail & Related papers  (2024-11-07T18:59:16Z)
• ICPL: Few-shot In-context Preference Learning via LLMs [15.84585737510038]
  We show that Large Language Models (LLMs) have native preference-learning capabilities that allow them to achieve sample-efficient preference learning. We propose In-Context Preference Learning (ICPL), which uses in-context learning capabilities of LLMs to reduce human query inefficiency.
  arXiv  Detail & Related papers  (2024-10-22T17:53:34Z)
• Skill-aware Mutual Information Optimisation for Generalisation in Reinforcement Learning [14.62474759939562]
  We introduce Skill-aware Mutual Information (SaMI), an optimisation objective that aids in distinguishing context embeddings according to skills. We then propose Skill-aware Noise Contrastive Estimation (SaNCE), a $K$-sample estimator used to optimise the SaMI objective. We empirically find that RL agents that learn by maximising SaMI achieve substantially improved zero-shot generalisation to unseen tasks.
  arXiv  Detail & Related papers  (2024-06-07T10:35:29Z)
• Agentic Skill Discovery [19.5703917813767]
  Language-conditioned robotic skills make it possible to apply the high-level reasoning of Large Language Models (LLMs) to low-level robotic control. A remaining challenge is to acquire a diverse set of fundamental skills. We introduce a novel framework for skill discovery that is entirely driven by LLMs.
  arXiv  Detail & Related papers  (2024-05-23T19:44:03Z)
• Acquiring Diverse Skills using Curriculum Reinforcement Learning with Mixture of Experts [58.220879689376744]
  Reinforcement learning (RL) is a powerful approach for acquiring a good-performing policy. We propose textbfDiverse textbfSkill textbfLearning (Di-SkilL) for learning diverse skills. We show on challenging robot simulation tasks that Di-SkilL can learn diverse and performant skills.
  arXiv  Detail & Related papers  (2024-03-11T17:49:18Z)
• Eureka: Human-Level Reward Design via Coding Large Language Models [121.91007140014982]
  Large Language Models (LLMs) have excelled as high-level semantic planners for sequential decision-making tasks. We present Eureka, a human-level reward design algorithm powered by LLMs. Eureka exploits the remarkable zero-shot generation, code-writing, and in-context improvement capabilities of state-of-the-art LLMs.
  arXiv  Detail & Related papers  (2023-10-19T17:31:01Z)
• Bootstrap Your Own Skills: Learning to Solve New Tasks with Large Language Model Guidance [66.615355754712]
  BOSS learns to accomplish new tasks by performing "skill bootstrapping" We demonstrate through experiments in realistic household environments that agents trained with our LLM-guided bootstrapping procedure outperform those trained with naive bootstrapping.
  arXiv  Detail & Related papers  (2023-10-16T02:43:47Z)
• C$\cdot$ASE: Learning Conditional Adversarial Skill Embeddings for Physics-based Characters [49.83342243500835]
  We present C$cdot$ASE, an efficient framework that learns conditional Adversarial Skill Embeddings for physics-based characters. C$cdot$ASE divides the heterogeneous skill motions into distinct subsets containing homogeneous samples for training a low-level conditional model. The skill-conditioned imitation learning naturally offers explicit control over the character's skills after training.
  arXiv  Detail & Related papers  (2023-09-20T14:34:45Z)
• Language Reward Modulation for Pretraining Reinforcement Learning [61.76572261146311]
  We propose leveraging the capabilities of LRFs as a pretraining signal for reinforcement learning. Our VLM pretraining approach, which is a departure from previous attempts to use LRFs, can warmstart sample-efficient learning on robot manipulation tasks.
  arXiv  Detail & Related papers  (2023-08-23T17:37:51Z)
• Residual Skill Policies: Learning an Adaptable Skill-based Action Space for Reinforcement Learning for Robotics [18.546688182454236]
  Skill-based reinforcement learning (RL) has emerged as a promising strategy to leverage prior knowledge for accelerated robot learning. We propose accelerating exploration in the skill space using state-conditioned generative models. We validate our approach across four challenging manipulation tasks, demonstrating our ability to learn across task variations.
  arXiv  Detail & Related papers  (2022-11-04T02:42:17Z)
• Hierarchical Kickstarting for Skill Transfer in Reinforcement Learning [27.69559938165733]
  Practising and honing skills forms a fundamental component of how humans learn, yet artificial agents are rarely specifically trained to perform them. We investigate how skills can be incorporated into the training of reinforcement learning (RL) agents in complex environments. Our experiments show that learning with a prior knowledge of useful skills can significantly improve the performance of agents on complex problems.
  arXiv  Detail & Related papers  (2022-07-23T19:23:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.