Copiloting Autonomous Multi-Robot Missions: A Game-inspired Supervisory Control Interface

• URL: http://arxiv.org/abs/2204.06647v1
• Date: Wed, 13 Apr 2022 21:54:31 GMT
• Title: Copiloting Autonomous Multi-Robot Missions: A Game-inspired Supervisory Control Interface
• Abstract summary: We introduce a video game-inspired interface, an autonomous mission assistant, and test and deploy these using a heterogeneous multi-agent system. This work leads to improved human-supervisory control for a multi-agent system reducing overhead from application switching, task planning, execution, and verification.
• Score: 8.05901693109037
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Real-world deployment of new technology and capabilities can be daunting. The recent DARPA Subterranean (SubT) Challenge, for instance, aimed at the advancement of robotic platforms and autonomy capabilities in three one-year development pushes. While multi-agent systems are traditionally deployed in controlled and structured environments that allow for controlled testing (e.g., warehouses), the SubT challenge targeted various types of unknown underground environments that imposed the risk of robot loss in the case of failure. In this work, we introduce a video game-inspired interface, an autonomous mission assistant, and test and deploy these using a heterogeneous multi-agent system in challenging environments. This work leads to improved human-supervisory control for a multi-agent system reducing overhead from application switching, task planning, execution, and verification while increasing available exploration time with this human-autonomy teaming platform.

Related papers

• ETA: A New Agentic Paradigm for Embodied Tasks [76.25893120982523]
  We introduce the Embodied Task Agent (ETA), a new paradigm for extending digital agents into the physical world.<n>ETA centers the robot around a Planner that chooses one Tool call at a time, an Interface that controls execution, and a World that returns the result and a fresh observation.<n>ETA provides replaceable Planners, composable Tools and Skills, auditable memory, replayable trajectories, and common interfaces for simulation and real robots.
  arXiv  Detail & Related papers  (2026-08-04T16:56:09Z)
• RACAS: Controlling Diverse Robots With a Single Agentic System [26.695876900159373]
  We introduce RACAS (Robot-Agnostic Control via Agentic Systems), a cooperative agentic architecture in which three modules communicate exclusively through natural language to provide closed-loop robot control.<n> RACAS requires only a natural language description of the robot, a definition of available actions, and a task specification; no source code, model weights, or reward functions need to be modified to move between platforms.<n>We evaluate RACAS on several tasks using a wheeled ground robot, a recently published novel multi-jointed robotic limb, and an underwater vehicle. RACAS consistently solved all assigned tasks across these radically different platforms, demonstrating the potential
  arXiv  Detail & Related papers  (2026-03-05T19:25:01Z)
• Self-Sovereign Agent [75.57858796824469]
  Self-sovereign AI systems can economically sustain and extend their own operation without human involvement.<n>Recent advances in large language models and agent frameworks have substantially expanded agents' practical capabilities.
  arXiv  Detail & Related papers  (2026-03-04T03:11:26Z)
• Athena: An Autonomous Open-Hardware Tracked Rescue Robot Platform [1.4777718769290527]
  We present Athena, an open-hardware rescue ground robot research platform with four individually reconfigurable flippers.<n>A novel mounting solution using an industrial PU belt and tooth inserts allows the replacement and testing of different track profiles.<n>The manipulator with a maximum reach of 1.54m can be used to operate doors, valves, and other objects of interest.
  arXiv  Detail & Related papers  (2026-02-23T14:38:23Z)
• MOSAIC: Modular Scalable Autonomy for Intelligent Coordination of Heterogeneous Robotic Teams [10.664029115378813]
  We present a scalable autonomy framework for multi-robot scientific exploration using a unified mission abstraction.<n>The proposed framework enables robust, scalable multi-robot scientific exploration with limited operator intervention.
  arXiv  Detail & Related papers  (2026-01-30T14:46:15Z)
• Fauna Sprout: A lightweight, approachable, developer-ready humanoid robot [3.3770931434110345]
  Sprout is a lightweight form factor with compliant control, limited joint torques, and soft exteriors to support safe operation in shared human spaces.<n>The platform integrates whole-body control, manipulation with integrated grippers, and virtual-reality-based teleoperation within a unified hardware-software stack.
  arXiv  Detail & Related papers  (2026-01-26T21:04:01Z)
• Heterogeneous Robot Collaboration in Unstructured Environments with Grounded Generative Intelligence [54.91177026001217]
  Large language model (LLM)-enabled teaming methods typically assume well-structured and known environments.<n>We present SPINE-HT, a framework that addresses these limitations by grounding the reasoning abilities of LLMs in the context of a heterogeneous robot team.<n>Our framework achieves nearly twice the success rate compared to prior LLM-enabled heterogeneous teaming approaches.
  arXiv  Detail & Related papers  (2025-10-30T18:24:38Z)
• Distributed AI Agents for Cognitive Underwater Robot Autonomy [5.644612398323221]
  This paper presents Underwater Robot Self-Organizing Autonomy (UROSA)<n>UROSA is a groundbreaking architecture leveraging distributed Large Language Model AI agents integrated within the Robot Operating System 2 (ROS 2) framework.<n>Central innovations include flexible agents dynamically adapting their roles, retrieval-augmented generation, and autonomous on-the-fly ROS 2 node generation.
  arXiv  Detail & Related papers  (2025-07-31T17:18:55Z)
• From Virtual Agents to Robot Teams: A Multi-Robot Framework Evaluation in High-Stakes Healthcare Context [2.016235597066821]
  Current frameworks treat agents as conceptual task executors rather than physically embodied entities.<n>We propose three design guidelines emphasizing process transparency, proactive failure recovery, and contextual grounding.<n>Our work informs the development of more resilient and robust multi-agent robotic systems.
  arXiv  Detail & Related papers  (2025-06-04T04:05:38Z)
• RoboRAN: A Unified Robotics Framework for Reinforcement Learning-Based Autonomous Navigation [15.548637925166986]
  We present a multi-domain framework for training, evaluating and deploying RL-based navigation policies across diverse robotic platforms and operational environments.<n>Our work presents four key contributions: (1) a scalable and modular framework, facilitating seamless robot-task interchangeability and reproducible training pipelines; (2) sim-to-real transfer demonstrated through real-world experiments with multiple robots; and (3) the release of the first open-source API for deploying Isaac Lab-trained policies to real robots.
  arXiv  Detail & Related papers  (2025-05-20T15:48:23Z)
• Deploying Foundation Model-Enabled Air and Ground Robots in the Field: Challenges and Opportunities [65.98704516122228]
  The integration of foundation models (FMs) into robotics has enabled robots to understand natural language and reason about the semantics in their environments.<n>This paper addresses the deployment of FM-enabled robots in the field, where missions often require a robot to operate in large-scale and unstructured environments.<n>We present the first demonstration of large-scale LLM-enabled robot planning in unstructured environments with several kilometers of missions.
  arXiv  Detail & Related papers  (2025-05-14T15:28:43Z)
• Multi-Task Interactive Robot Fleet Learning with Visual World Models [25.001148860168477]
  Sirius-Fleet is a multi-task interactive robot fleet learning framework. It monitors robot performance during deployment and involves humans to correct the robot's actions when necessary. As the robot autonomy improves, anomaly predictors automatically adapt their prediction criteria.
  arXiv  Detail & Related papers  (2024-10-30T04:49:39Z)
• EMOS: Embodiment-aware Heterogeneous Multi-robot Operating System with LLM Agents [33.77674812074215]
  We introduce a novel multi-agent framework designed to enable effective collaboration among heterogeneous robots. We propose a self-prompted approach, where agents comprehend robot URDF files and call robot kinematics tools to generate descriptions of their physics capabilities. The Habitat-MAS benchmark is designed to assess how a multi-agent framework handles tasks that require embodiment-aware reasoning.
  arXiv  Detail & Related papers  (2024-10-30T03:20:01Z)
• GRAPPA: Generalizing and Adapting Robot Policies via Online Agentic Guidance [15.774237279917594]
  We propose an agentic framework for robot self-guidance and self-improvement. Our framework iteratively grounds a base robot policy to relevant objects in the environment. We demonstrate that our approach can effectively guide manipulation policies to achieve significantly higher success rates.
  arXiv  Detail & Related papers  (2024-10-09T02:00:37Z)
• Commonsense Reasoning for Legged Robot Adaptation with Vision-Language Models [81.55156507635286]
  Legged robots are physically capable of navigating a diverse variety of environments and overcoming a wide range of obstructions. Current learning methods often struggle with generalization to the long tail of unexpected situations without heavy human supervision. We propose a system, VLM-Predictive Control (VLM-PC), combining two key components that we find to be crucial for eliciting on-the-fly, adaptive behavior selection.
  arXiv  Detail & Related papers  (2024-07-02T21:00:30Z)
• RoboScript: Code Generation for Free-Form Manipulation Tasks across Real and Simulation [77.41969287400977]
  This paper presents textbfRobotScript, a platform for a deployable robot manipulation pipeline powered by code generation. We also present a benchmark for a code generation benchmark for robot manipulation tasks in free-form natural language. We demonstrate the adaptability of our code generation framework across multiple robot embodiments, including the Franka and UR5 robot arms.
  arXiv  Detail & Related papers  (2024-02-22T15:12:00Z)
• Reinforcement Learning for Versatile, Dynamic, and Robust Bipedal Locomotion Control [106.32794844077534]
  This paper presents a study on using deep reinforcement learning to create dynamic locomotion controllers for bipedal robots. We develop a general control solution that can be used for a range of dynamic bipedal skills, from periodic walking and running to aperiodic jumping and standing. This work pushes the limits of agility for bipedal robots through extensive real-world experiments.
  arXiv  Detail & Related papers  (2024-01-30T10:48:43Z)
• AutoRT: Embodied Foundation Models for Large Scale Orchestration of Robotic Agents [109.3804962220498]
  AutoRT is a system to scale up the deployment of operational robots in completely unseen scenarios with minimal human supervision. We demonstrate AutoRT proposing instructions to over 20 robots across multiple buildings and collecting 77k real robot episodes via both teleoperation and autonomous robot policies. We experimentally show that such "in-the-wild" data collected by AutoRT is significantly more diverse, and that AutoRT's use of LLMs allows for instruction following data collection robots that can align to human preferences.
  arXiv  Detail & Related papers  (2024-01-23T18:45:54Z)
• Human-Machine Teaming for UAVs: An Experimentation Platform [6.809734620480709]
  We present the Cogment human-machine teaming experimentation platform. It implements human-machine teaming (HMT) use cases that can involve learning AI agents, static AI agents, and humans. We hope to facilitate further research on human-machine teaming in critical systems and defense environments.
  arXiv  Detail & Related papers  (2023-12-18T21:35:02Z)
• Autonomous Aerial Robot for High-Speed Search and Intercept Applications [86.72321289033562]
  A fully-autonomous aerial robot for high-speed object grasping has been proposed. As an additional sub-task, our system is able to autonomously pierce balloons located in poles close to the surface. Our approach has been validated in a challenging international competition and has shown outstanding results.
  arXiv  Detail & Related papers  (2021-12-10T11:49:51Z)
• SABER: Data-Driven Motion Planner for Autonomously Navigating Heterogeneous Robots [112.2491765424719]
  We present an end-to-end online motion planning framework that uses a data-driven approach to navigate a heterogeneous robot team towards a global goal. We use model predictive control (SMPC) to calculate control inputs that satisfy robot dynamics, and consider uncertainty during obstacle avoidance with chance constraints. recurrent neural networks are used to provide a quick estimate of future state uncertainty considered in the SMPC finite-time horizon solution. A Deep Q-learning agent is employed to serve as a high-level path planner, providing the SMPC with target positions that move the robots towards a desired global goal.
  arXiv  Detail & Related papers  (2021-08-03T02:56:21Z)
• Autonomous Intruder Detection Using a ROS-Based Multi-Robot System Equipped with 2D-LiDAR Sensors [0.5512295869673147]
  This paper proposes a multi-robot system for intruder detection in a single-range-sensor-per-robot scenario with centralized processing of detections from all robots by our central bot MIDNet. This work is aimed at providing an autonomous multi-robot security solution for a warehouse in the absence of human personnel.
  arXiv  Detail & Related papers  (2020-11-07T19:49:07Z)
• Co-Evolution of Multi-Robot Controllers and Task Cues for Off-World Open Pit Mining [0.6091702876917281]
  This paper presents a novel method for developing scalable controllers for use in multi-robot excavation and site-preparation scenarios. The controller starts with a blank slate and does not require human-authored operations scripts nor detailed modeling of the kinematics and dynamics of the excavator. In this paper, we explore the use of templates and task cues to improve group performance further and minimize antagonism.
  arXiv  Detail & Related papers  (2020-09-19T03:13:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.