SkillComposer: Learning Reusable Skills for Natural-Language Robot Programming

• URL: http://arxiv.org/abs/2608.14944v1
• Date: Fri, 14 Aug 2026 23:49:54 GMT
• Title: SkillComposer: Learning Reusable Skills for Natural-Language Robot Programming
• Abstract summary: SkillComposer is an interactive natural-language robot programming system for simulation environments.<n>Successful programs are stored and processed by an online library-learning algorithm.<n>Results show that evaluator-guided generation and learned abstractions improve success rates and usability.
• Score: 15.199350753007339
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Natural-language interfaces can lower the barrier to programming robots, but existing systems struggle when users request complex tasks. While large language models (LLMs) perform well with simple commands, they often struggle to generate code for multi-step tasks, decompose high-level instructions, or reuse prior solutions. We present SkillComposer, an interactive natural-language robot programming system for simulation environments that continually learns reusable program abstractions. SkillComposer uses a generate-test architecture in which an LLM iteratively generates and revises robot programs before execution. Successful programs are stored and processed by an online library-learning algorithm that compresses recurring function sequences into reusable macro skills for future tasks. We evaluate SkillComposer through ablation experiments and a user study with 12 participants to determine its effectiveness on manipulation and robot caregiving tasks. The results show that evaluator-guided generation and learned abstractions improve success rates and usability while reducing user effort in natural-language robot programming.

Related papers

• RoBridge: A Hierarchical Architecture Bridging Cognition and Execution for General Robotic Manipulation [90.81956345363355]
  RoBridge is a hierarchical intelligent architecture for general robotic manipulation.<n>It consists of a high-level cognitive planner (HCP) based on a large-scale pre-trained vision-language model (VLM)<n>It unleashes the procedural skill of reinforcement learning, effectively bridging the gap between cognition and execution.
  arXiv  Detail & Related papers  (2025-05-03T06:17:18Z)
• Towards No-Code Programming of Cobots: Experiments with Code Synthesis by Large Code Models for Conversational Programming [17.22551679328024]
  Large Language Models (LLMs) are designed to do in-context learning for conversational code generation.<n>We evaluate the capabilities of state-of-the-art LLMs for synthesising this kind of code, given in-context examples.
  arXiv  Detail & Related papers  (2024-09-17T10:04:50Z)
• Interactive Planning Using Large Language Models for Partially Observable Robotics Tasks [54.60571399091711]
  Large Language Models (LLMs) have achieved impressive results in creating robotic agents for performing open vocabulary tasks. We present an interactive planning technique for partially observable tasks using LLMs.
  arXiv  Detail & Related papers  (2023-12-11T22:54:44Z)
• RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulation [68.70755196744533]
  RoboGen is a generative robotic agent that automatically learns diverse robotic skills at scale via generative simulation. Our work attempts to extract the extensive and versatile knowledge embedded in large-scale models and transfer them to the field of robotics.
  arXiv  Detail & Related papers  (2023-11-02T17:59:21Z)
• Learning a Hierarchical Planner from Humans in Multiple Generations [21.045112705349222]
  We present natural programming, a library learning system that combines programmatic learning with a hierarchical planner. A user teaches the system via curriculum building, by identifying a challenging yet not impossible goal. The system solves for the goal via hierarchical planning, using the linguistic hints to guide its probability distribution.
  arXiv  Detail & Related papers  (2023-10-17T22:28:13Z)
• Incremental Learning of Humanoid Robot Behavior from Natural Interaction and Large Language Models [23.945922720555146]
  We propose a system to achieve incremental learning of complex behavior from natural interaction. We integrate the system in the robot cognitive architecture of the humanoid robot ARMAR-6.
  arXiv  Detail & Related papers  (2023-09-08T13:29:05Z)
• Language to Rewards for Robotic Skill Synthesis [37.21434094015743]
  We introduce a new paradigm that harnesses large language models (LLMs) to define reward parameters that can be optimized and accomplish variety of robotic tasks. Using reward as the intermediate interface generated by LLMs, we can effectively bridge the gap between high-level language instructions or corrections to low-level robot actions.
  arXiv  Detail & Related papers  (2023-06-14T17:27:10Z)
• What Matters in Language Conditioned Robotic Imitation Learning [26.92329260907805]
  We study the most critical challenges in learning language conditioned policies from offline free-form imitation datasets. We present a novel approach that significantly outperforms the state of the art on the challenging language conditioned long-horizon robot manipulation CALVIN benchmark.
  arXiv  Detail & Related papers  (2022-04-13T08:45:32Z)
• Lifelong Robotic Reinforcement Learning by Retaining Experiences [61.79346922421323]
  Many multi-task reinforcement learning efforts assume the robot can collect data from all tasks at all times. In this work, we study a practical sequential multi-task RL problem motivated by the practical constraints of physical robotic systems. We derive an approach that effectively leverages the data and policies learned for previous tasks to cumulatively grow the robot's skill-set.
  arXiv  Detail & Related papers  (2021-09-19T18:00:51Z)
• Learning Adaptive Language Interfaces through Decomposition [89.21937539950966]
  We introduce a neural semantic parsing system that learns new high-level abstractions through decomposition. Users interactively teach the system by breaking down high-level utterances describing novel behavior into low-level steps.
  arXiv  Detail & Related papers  (2020-10-11T08:27:07Z)
• Scalable Multi-Task Imitation Learning with Autonomous Improvement [159.9406205002599]
  We build an imitation learning system that can continuously improve through autonomous data collection. We leverage the robot's own trials as demonstrations for tasks other than the one that the robot actually attempted. In contrast to prior imitation learning approaches, our method can autonomously collect data with sparse supervision for continuous improvement.
  arXiv  Detail & Related papers  (2020-02-25T18:56:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.