MistyPilot: Enabling Social-Robot Control through Multi-Agent LLM Skill Orchestration

• URL: http://arxiv.org/abs/2608.15549v1
• Date: Sun, 16 Aug 2026 05:45:16 GMT
• Title: MistyPilot: Enabling Social-Robot Control through Multi-Agent LLM Skill Orchestration
• Abstract summary: We present MistyPilot, a multi-agent framework that orchestrates the corresponding skills on the Misty social robot.<n>A Task Router dispatches each instruction to one of two specialized agents.<n>MistyPilot attains high accuracy on routing, sensor-skill binding, task-state parsing, result reuse, and skill extension up to 100 skills.
• Score: 18.20564232594413
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Programming small social robots from natural-language instructions requires more than invoking isolated APIs. Interactive tasks combine reactive physical behaviors with stateful social behaviors, while existing interfaces often require developers to manually compose APIs into skills, configure their parameters, bind sensor events to skills, and manage task states at runtime. We present MistyPilot, a multi-agent LLM framework that interprets high-level natural-language instructions and orchestrates the corresponding skills on the Misty social robot. A Task Router dispatches each instruction to one of two specialized agents: a Physically Interactive Agent for sensor-triggered robot control and direct skill invocation, and a Social Interaction Agent for dialogue-oriented task-state management and context-dependent multimodal response generation. To improve efficiency, the Social Interaction Agent reuses previously generated results when applicable and invokes full generation otherwise. We evaluate MistyPilot on five component-level suites, with sensor bindings and skill invocations executed on the physical Misty robot, and a preliminary user study with 12 participants. MistyPilot attains high accuracy on routing, sensor-skill binding, task-state parsing, result reuse, and skill extension up to 100 skills, and lower variance than an otherwise identical single-agent baseline, while participants report positive perceptions of usability and interaction quality. The code will be made publicly available via the project page.

Related papers

• Ludi${}_{\scriptscriptstyle 0.1}$: An Agentic System for Socially Intelligent Robots [22.26258894927344]
  Ludi$_scriptscriptstyle 0.1$ is an agentic system for socially intelligent robots.<n>It integrates interactive speech, multimodal reasoning, memory, navigation, and learned manipulation.<n>Ludi$_scriptstyle 0.1$ demonstrates a practical path toward fluid human-robot collaboration.
  arXiv  Detail & Related papers  (2026-08-22T16:38:59Z)
• Playful Agentic Robot Learning [108.08954330298549]
  We study Playful Agentic Robot Learning, where an embodied coding agent uses self-directed play as a continual skill-learning stage before downstream tasks arrive.<n>During play, RATs proposes novel yet learnable exploratory tasks, plans and executes robot-code policies, verifies intermediate progress, diagnoses failures, retries with dense, step-level feedback, and distills successful executions into a persistent code skill library.
  arXiv  Detail & Related papers  (2026-06-17T17:55:23Z)
• ComPASS: Towards Personalized Agentic Social Support via Tool-Augmented Companionship [55.402028219661986]
  We develop tools to empower agents with external tools to execute diverse actions.<n>We first design a dozen user-centric tools simulating various multimedia applications, which can cover different types of social support behaviors.<n>Based on ComPASS-Bench, we further synthesize tool use records to fine-tune the Qwen3-8B model, yielding a task-specific ComPASS-Qwen.
  arXiv  Detail & Related papers  (2026-04-20T14:49:16Z)
• WebXSkill: Skill Learning for Autonomous Web Agents [104.76374637691212]
  WebXSkill is a framework that bridges the gap between code-based skills and natural language guidance.<n>On WebArena and WebVoyager, WebXSkill improves task success rate by up to 9.8 and 12.9 points over the baseline.
  arXiv  Detail & Related papers  (2026-04-14T21:48:15Z)
• MistyPilot: An Agentic Fast-Slow Thinking LLM Framework for Misty Social Robots [19.141461555104843]
  MistyPilot is an agentic LLM-driven framework for autonomous tool selection, orchestration, and parameter configuration.<n>MistyPilot comprises two core components: a Physically Interactive Agent (PIA) and a Socially Intelligent Agent (SIA)
  arXiv  Detail & Related papers  (2026-03-04T02:00:43Z)
• RoboOmni: Proactive Robot Manipulation in Omni-modal Context [165.09049429566238]
  We introduce cross-modal contextual instructions, a new setting where intent is derived from spoken dialogue, environmental sounds, and visual cues rather than explicit commands.<n>We present RoboOmni, a framework based on end-to-end omni-modal LLMs that unifies intention recognition, interaction confirmation, and action execution.<n>Experiments in simulation and real-world settings show that Robo Omni surpasses text- and ASR-based baselines in success rate, inference speed, intention recognition, and proactive assistance.
  arXiv  Detail & Related papers  (2025-10-27T18:49:03Z)
• Interpretable Robot Control via Structured Behavior Trees and Large Language Models [0.14990005092937678]
  This paper presents a novel framework that bridges natural language understanding and robotic execution.<n>The proposed approach is practical in real-world scenarios, with an average cognition-to-execution accuracy of approximately 94%.
  arXiv  Detail & Related papers  (2025-08-13T08:53:13Z)
• Breaking Single-Tester Limits: Multi-Agent LLMs for Multi-User Feature Testing [15.383375235673954]
  We propose MAdroid, a novel multi-agent approach powered by the Large Language Models (LLMs) to automate the multi-user interactive task for app feature testing.<n>Specifically, MAdroid employs two functional types of multi-agents: user agents (Operator) and supervisor agents (Coordinator and Observer)<n>Our evaluation, which included 41 multi-user interactive tasks, demonstrates the effectiveness of our approach, achieving 82.9% of the tasks with 96.8% action similarity.
  arXiv  Detail & Related papers  (2025-06-21T01:38:53Z)
• ROS-LLM: A ROS framework for embodied AI with task feedback and structured reasoning [74.58666091522198]
  We present a framework for intuitive robot programming by non-experts. We leverage natural language prompts and contextual information from the Robot Operating System (ROS) Our system integrates large language models (LLMs), enabling non-experts to articulate task requirements to the system through a chat interface.
  arXiv  Detail & Related papers  (2024-06-28T08:28:38Z)
• RoboScript: Code Generation for Free-Form Manipulation Tasks across Real and Simulation [77.41969287400977]
  This paper presents textbfRobotScript, a platform for a deployable robot manipulation pipeline powered by code generation. We also present a benchmark for a code generation benchmark for robot manipulation tasks in free-form natural language. We demonstrate the adaptability of our code generation framework across multiple robot embodiments, including the Franka and UR5 robot arms.
  arXiv  Detail & Related papers  (2024-02-22T15:12:00Z)
• Chat with the Environment: Interactive Multimodal Perception Using Large Language Models [19.623070762485494]
  Large Language Models (LLMs) have shown remarkable reasoning ability in few-shot robotic planning. Our study demonstrates that LLMs can provide high-level planning and reasoning skills and control interactive robot behavior in a multimodal environment.
  arXiv  Detail & Related papers  (2023-03-14T23:01:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.