CLASP: Language-Driven Robot Skill Selection and Composition using Task-Parameterized Learning

• URL: http://arxiv.org/abs/2606.08169v1
• Date: Sat, 06 Jun 2026 13:33:39 GMT
• Title: CLASP: Language-Driven Robot Skill Selection and Composition using Task-Parameterized Learning
• Abstract summary: Foundation models such as vision-language-action (VLA) and vision-language models (VLMs) provide intuitive interaction channels but require extensive data.<n>This work bridges this gap through a modular architecture combining task- parameterized kernelized movement primitives (TP-KMPs) with pretrained VLMs.<n>During learning, skills are acquired from 2 to 5 kinesthetic demonstrations, and the VLM generates skill schemas describing each skill's parameters and preconditions.<n>During execution, the VLM interprets commands to select skills, reason about parameter bindings, and create novel behaviors through co
• Score: 9.66356526923778
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Enabling robots to understand and execute tasks from natural language commands while maintaining data efficiency remains challenging. Foundation models such as vision-language-action (VLA) and vision-language models (VLMs) provide intuitive interaction channels but require extensive data; task-parameterized imitation learning achieves data efficiency but lacks natural language grounding. This work bridges this gap through a modular architecture combining task-parameterized kernelized movement primitives (TP-KMPs) with pretrained VLMs. During learning, skills are acquired from 2 to 5 kinesthetic demonstrations, and the VLM generates skill schemas describing each skill's parameters and preconditions. During execution, the VLM interprets commands to select skills, reason about parameter bindings, and create novel behaviors through covariance-weighted composition. When no skill or composition suffices, the system identifies capability gaps and requests targeted demonstrations, all without fine-tuning. Validation on a 7-DoF manipulator shows success rates of 73.3%-100% in scenarios requiring skill selection, composition, and active learning.

Related papers

• Hierarchical Skill Retrieval for Data-Efficient Adaptation of Vision-Language-Action Models [10.94594187411586]
  Hierarchical Skill Retrieval (HSR) is a retrieval framework for data-efficient VLA adaptation.<n>We show HSR improves the average success rate by 10.3% and 21.3% over the strongest baseline.
  arXiv  Detail & Related papers  (2026-08-25T04:04:49Z)
• Decoupling the Declarative from the Procedural in Vision-Language-Action Models [65.22639791239932]
  Policy trained to clone a behavior from object-specific demonstrations must generalize beyond that object.<n>We propose w$2$VLA, a new Vision-Language-Action model with restructured information flow.<n>Unlike popular, state-of-the-art VLAs, we show that our modular approach successfully decouples knowledge representations.
  arXiv  Detail & Related papers  (2026-06-19T14:43:58Z)
• Two Bridges, One Pathway: From VLMs to Generalizable VLAs with Embodied Trajectory-Coupled Data [93.7685703383343]
  Vision-language models (VLMs) are powerful general-purpose reasoners, yet converting them into robot control policies is surprisingly difficult.<n>We argue that this gap can be bridged gradually with the right intermediate data.<n>We show that mixing task-relevant out-of-distribution ETC data with a small amount of action data enables the model to generalize to novel visual-language conditions.
  arXiv  Detail & Related papers  (2026-06-07T08:57:51Z)
• Lifelong Learning in Vision-Language Models: Enhanced EWC with Cross-Modal Knowledge Retention [0.0]
  Large language-vision models (LVLMs) have revolutionized AI by enabling understanding across textual and visual modalities.<n>These models excel at tasks like image captioning, visual question answering, and cross-modal retrieval.<n>This paper presents a comprehensive continual learning framework for LVLMs that combines enhanced Elastic Weight Consolidation with parameter-efficient fine-tuning techniques.
  arXiv  Detail & Related papers  (2026-05-12T22:05:30Z)
• Mechanistic Finetuning of Vision-Language-Action Models via Few-Shot Demonstrations [76.79742393097358]
  Vision-Language Action (VLAs) models promise to extend the remarkable success of vision-language models (VLMs) to robotics.<n>Existing fine-tuning methods lack specificity, adapting the same set of parameters regardless of a task's visual, linguistic, and physical characteristics.<n>Inspired by functional specificity in neuroscience, we hypothesize that it is more effective to finetune sparse model representations specific to a given task.
  arXiv  Detail & Related papers  (2025-11-27T18:50:21Z)
• SPECI: Skill Prompts based Hierarchical Continual Imitation Learning for Robot Manipulation [3.1997825444285457]
  Real-world robot manipulation in dynamic unstructured environments requires lifelong adaptability to evolving objects, scenes and tasks.<n>Traditional imitation learning relies on static training paradigms, which are ill-suited for lifelong adaptation.<n>We propose Skill Prompts-based HiErarchical Continual Imitation Learning (SPECI), a novel end-to-end hierarchical CIL policy architecture for robot manipulation.
  arXiv  Detail & Related papers  (2025-04-22T03:30:38Z)
• GROVE: A Generalized Reward for Learning Open-Vocabulary Physical Skill [25.686589649523587]
  Learning open-vocabulary physical skills for simulated agents presents a significant challenge in artificial intelligence.<n>We introduce GROVE, a generalized reward framework that enables open-vocabulary physical skill learning without manual engineering or task-specific demonstrations.<n>To bridge the domain gap between simulation and natural images, we develop Pose2CLIP, a lightweight mapper that efficiently projects agent poses directly into semantic feature space.
  arXiv  Detail & Related papers  (2025-04-05T14:44:47Z)
• LLaRA: Supercharging Robot Learning Data for Vision-Language Policy [56.505551117094534]
  We introduce LLaRA: Large Language and Robotics Assistant, a framework that formulates robot action policy as visuo-textual conversations.<n>First, we present an automated pipeline to generate conversation-style instruction tuning data for robots from existing behavior cloning datasets.<n>We show that a VLM finetuned with a limited amount of such datasets can produce meaningful action decisions for robotic control.
  arXiv  Detail & Related papers  (2024-06-28T17:59:12Z)
• SkillDiffuser: Interpretable Hierarchical Planning via Skill Abstractions in Diffusion-Based Task Execution [75.2573501625811]
  Diffusion models have demonstrated strong potential for robotic trajectory planning. generating coherent trajectories from high-level instructions remains challenging. We propose SkillDiffuser, an end-to-end hierarchical planning framework.
  arXiv  Detail & Related papers  (2023-12-18T18:16:52Z)
• Interactive Planning Using Large Language Models for Partially Observable Robotics Tasks [54.60571399091711]
  Large Language Models (LLMs) have achieved impressive results in creating robotic agents for performing open vocabulary tasks. We present an interactive planning technique for partially observable tasks using LLMs.
  arXiv  Detail & Related papers  (2023-12-11T22:54:44Z)
• Skill-based Multi-objective Reinforcement Learning of Industrial Robot Tasks with Planning and Knowledge Integration [0.4949816699298335]
  We introduce an approach that provides a combination of task-level planning with targeted learning of scenario-specific parameters for skill-based systems. We demonstrate the efficacy and versatility of our approach by learning skill parameters for two different contact-rich tasks.
  arXiv  Detail & Related papers  (2022-03-18T16:03:27Z)
• HyperPELT: Unified Parameter-Efficient Language Model Tuning for Both Language and Vision-and-Language Tasks [38.43269863509866]
  How to perform parameter-efficient fine-tuning has become fairly important for quick transfer learning and deployment. We design a novel unified parameter-efficient transfer learning framework that works effectively on both pure language and V&L tasks. Our proposed framework adds fewer trainable parameters in multi-task learning while achieving superior performances and transfer ability compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2022-03-08T06:51:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.