MOMO: A framework for seamless physical, verbal, and graphical robot skill learning and adaptation

• URL: http://arxiv.org/abs/2604.20468v2
• Date: Thu, 23 Apr 2026 12:18:52 GMT
• Title: MOMO: A framework for seamless physical, verbal, and graphical robot skill learning and adaptation
• Abstract summary: We present an interactive framework that enables robot skill adaptation through three complementary modalities.<n>The framework integrates five components: energy-based human-intention detection, a tool-based LLM architecture, and Kernelized Movement Primitives.<n>We demonstrate that this tool-based LLM architecture generalizes skill adaptation from KMPs to ergodic control, enabling voice-commanded surface finishing.
• Score: 8.512341071836513
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Industrial robot applications require increasingly flexible systems that non-expert users can easily adapt for varying tasks and environments. However, different adaptations benefit from different interaction modalities. We present an interactive framework that enables robot skill adaptation through three complementary modalities: kinesthetic touch for precise spatial corrections, natural language for high-level semantic modifications, and a graphical web interface for visualizing geometric relations and trajectories, inspecting and adjusting parameters, and editing via-points by drag-and-drop. The framework integrates five components: energy-based human-intention detection, a tool-based LLM architecture (where the LLM selects and parameterizes predefined functions rather than generating code) for safe natural language adaptation, Kernelized Movement Primitives (KMPs) for motion encoding, probabilistic Virtual Fixtures for guided demonstration recording, and ergodic control for surface finishing. We demonstrate that this tool-based LLM architecture generalizes skill adaptation from KMPs to ergodic control, enabling voice-commanded surface finishing. Validation on a 7-DoF torque-controlled robot at the Automatica 2025 trade fair demonstrates the practical applicability of our approach in industrial settings.

Related papers

• A Continual Learning Framework for Adaptive Control of Modular Soft Robots [4.374184567811783]
  Soft robots have attracted significant attention in applications such as medical intervention, rehabilitation, and robotic manipulation.<n>MSRs represent an emerging class of robotic systems with highly deformable and reconfigurable structures capable of performing complex tasks.<n>Existing approaches typically require controllers to be retrained from scratch whenever the robot morphology changes.<n>We propose a continual learning inspired control framework capable of incrementally adapting to changes in robot morphology.
  arXiv  Detail & Related papers  (2026-07-07T19:11:07Z)
• In-Context Model Predictive Generation: Open-Vocabulary Motion Synthesis from Language Models to Physics [109.99871326497333]
  In-Context Model Predictive Generation (ICMPG) is a framework that integrates language-model planning with inference-time physical feedback.<n>ICMPG reformulates motion synthesis as a Model Predictive Control (MPC)-like process with two modules.
  arXiv  Detail & Related papers  (2026-06-25T12:50:33Z)
• Integrating Large Language Model Agents with Digital Twins for Industrial Autonomous Systems [1.4101164066938148]
  Modern production environments require greater adaptability, faster reconfiguration, and more intuitive human-machine interaction.<n>Traditional rule-based systems rely on fixed logic and cannot autonomously adapt to changing conditions.<n>This dissertation proposes a three-layer framework that integrates large language models (LLMs), digital twins, and automation systems into an autonomous system.
  arXiv  Detail & Related papers  (2026-06-18T09:48:44Z)
• Learning Context-Aware Neural ODE Dynamics for Adaptive Robotic Control [55.01272784312405]
  We propose a context-aware dynamics model based on neural ordinary differential equations.<n>We validate the approach across diverse robotic platforms, including a quadrotor in simulation, as well as a Sphero BOLT robot and a Fanuc manipulator in real-world experiments.
  arXiv  Detail & Related papers  (2026-06-13T20:55:46Z)
• IROSA: Interactive Robot Skill Adaptation using Natural Language [9.66356526923778]
  We present a novel framework that enables open-vocabulary skill adaptation through a tool-based architecture.<n>We demonstrate the framework on a 7-DoF torque-controlled robot performing an industrial bearing ring insertion task.
  arXiv  Detail & Related papers  (2026-03-04T09:54:09Z)
• Executable Analytic Concepts as the Missing Link Between VLM Insight and Precise Manipulation [70.8381970762877]
  Vision-Language Models (VLMs) have demonstrated remarkable capabilities in semantic reasoning and task planning.<n>We introduce GRACE, a novel framework that grounds VLM-based reasoning through executable analytic concepts.<n>G GRACE provides a unified and interpretable interface between high-level instruction understanding and low-level robot control.
  arXiv  Detail & Related papers  (2025-10-09T09:08:33Z)
• SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and Object Manipulation [50.060274413294586]
  We introduce the concept of semantic orientation, which defines object orientations using natural language.<n>By integrating semantic orientation into VLM agents, our SoFar framework enables 6-DoF spatial reasoning and generates robotic actions.
  arXiv  Detail & Related papers  (2025-02-18T18:59:02Z)
• Flex: End-to-End Text-Instructed Visual Navigation from Foundation Model Features [59.892436892964376]
  We investigate the minimal data requirements and architectural adaptations necessary to achieve robust closed-loop performance with vision-based control policies.<n>Our findings are synthesized in Flex (Fly lexically), a framework that uses pre-trained Vision Language Models (VLMs) as frozen patch-wise feature extractors.<n>We demonstrate the effectiveness of this approach on a quadrotor fly-to-target task, where agents trained via behavior cloning successfully generalize to real-world scenes.
  arXiv  Detail & Related papers  (2024-10-16T19:59:31Z)
• SKT: Integrating State-Aware Keypoint Trajectories with Vision-Language Models for Robotic Garment Manipulation [82.61572106180705]
  This paper presents a unified approach using vision-language models (VLMs) to improve keypoint prediction across various garment categories. We created a large-scale synthetic dataset using advanced simulation techniques, allowing scalable training without extensive real-world data. Experimental results indicate that the VLM-based method significantly enhances keypoint detection accuracy and task success rates.
  arXiv  Detail & Related papers  (2024-09-26T17:26:16Z)
• TLControl: Trajectory and Language Control for Human Motion Synthesis [68.09806223962323]
  We present TLControl, a novel method for realistic human motion synthesis. It incorporates both low-level Trajectory and high-level Language semantics controls. It is practical for interactive and high-quality animation generation.
  arXiv  Detail & Related papers  (2023-11-28T18:54:16Z)
• Language to Rewards for Robotic Skill Synthesis [37.21434094015743]
  We introduce a new paradigm that harnesses large language models (LLMs) to define reward parameters that can be optimized and accomplish variety of robotic tasks. Using reward as the intermediate interface generated by LLMs, we can effectively bridge the gap between high-level language instructions or corrections to low-level robot actions.
  arXiv  Detail & Related papers  (2023-06-14T17:27:10Z)
• Meta-Reinforcement Learning for Adaptive Motor Control in Changing Robot Dynamics and Environments [3.5309638744466167]
  This work developed a meta-learning approach that adapts the control policy on the fly to different changing conditions for robust locomotion. The proposed method constantly updates the interaction model, samples feasible sequences of actions of estimated the state-action trajectories, and then applies the optimal actions to maximize the reward.
  arXiv  Detail & Related papers  (2021-01-19T12:57:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.