HATS: A Human-Agent Teleoperation System for Multi-Arm Data Collection

• URL: http://arxiv.org/abs/2606.16491v1
• Date: Mon, 15 Jun 2026 09:56:24 GMT
• Title: HATS: A Human-Agent Teleoperation System for Multi-Arm Data Collection
• Abstract summary: HATS is a human-agent teleoperation system that enables a single human operator to collect data for multi-arm manipulation tasks.<n>Our system decouples the control space: two primary arms are directly teleoperated by the human, while two assistive arms are controlled by a training-free agent that handles sub-tasks.
• Score: 45.96541974744152
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Many real-world manipulation scenarios, such as handling complex collaborative tasks and dealing with large workspaces, require coordination of more than two robotic arms. Consequently, an effective multi-arm teleoperation system is required to collect demonstrations for training coordinated multi-arm manipulation policies. However, existing teleoperation frameworks mainly focus on single-operator or multi-operator setups, facing a practical trade-off between the cognitive load placed on a single operator and the coordination cost incurred by multiple operators. To address this problem, we introduce HATS, a human-agent teleoperation system that enables a single human operator, assisted by an MLLM-based agent, to collect data for multi-arm manipulation tasks. Our system decouples the control space: two primary arms are directly teleoperated by the human, while two assistive arms are controlled by a training-free agent that handles sub-tasks. In addition, the human operator can use voice commands to prevent collisions and correct assistive arm behaviors during execution. Extensive evaluations demonstrate that HATS achieves data collection efficiency and success rates comparable to expert dual-human teams. Moreover, downstream policy evaluations demonstrate the efficacy and quality of the data collected through HATS.

Related papers

• HAS-Bench: Evaluating LLM-Based Human-Agent Systems under Configurable Human Participation [54.200005792928344]
  We introduce a graph-based framework that represents humans and agents as first-class participants with explicit roles, permissions, communication paths, and action authority.<n>Building on this framework, HAS-Bench evaluates Human-Agent Systems under human participation across agency levels, interaction channels, and persona policies.<n> Experiments across six domains show that human participation can substantially improve task completion and failure recovery, but the gains depend on when, how, and by whom human input is exercised.
  arXiv  Detail & Related papers  (2026-07-05T14:23:27Z)
• CollabSim: A CSCW-Grounded Methodology for Investigating Collaborative Competence of LLM Agents through Controlled Multi-Agent Experiments [32.043112677406064]
  CollabSim is a simulation framework that combines a theory-grounded definition of collaborative capabilities, controlled manipulation of interaction conditions, and action-level probing of agents' internal states.<n> Experiments show that CollabSim can capture condition effects, separate model performance patterns, and reveal task-dependent effects of agent design.
  arXiv  Detail & Related papers  (2026-06-04T17:06:22Z)
• SynAgent: Generalizable Cooperative Humanoid Manipulation via Solo-to-Cooperative Agent Synergy [59.86199649493784]
  Controllable cooperative humanoid manipulation is a fundamental yet challenging problem for embodied intelligence.<n>We present SynAgent, a framework that enables scalable and physically plausible cooperative manipulation.<n>In experiments, SynAgent significantly outperforms existing baselines in both cooperative imitation and trajectory-conditioned control.
  arXiv  Detail & Related papers  (2026-04-20T17:46:20Z)
• HACTS: a Human-As-Copilot Teleoperation System for Robot Learning [47.9126187195398]
  We introduce HACTS (Human-As-Copilot Teleoperation System), a novel system that establishes bilateral, real-time joint synchronization between a robot arm and teleoperation hardware.<n>This simple yet effective feedback mechanism, akin to a steering wheel in autonomous vehicles, enables the human copilot to intervene seamlessly while collecting action-correction data for future learning.
  arXiv  Detail & Related papers  (2025-03-31T13:28:13Z)
• Human-Agent Joint Learning for Efficient Robot Manipulation Skill Acquisition [48.65867987106428]
  We introduce a novel system for joint learning between human operators and robots. It enables human operators to share control of a robot end-effector with a learned assistive agent. It reduces the need for human adaptation while ensuring the collected data is of sufficient quality for downstream tasks.
  arXiv  Detail & Related papers  (2024-06-29T03:37:29Z)
• CLAS: Coordinating Multi-Robot Manipulation with Central Latent Action Spaces [9.578169216444813]
  This paper proposes an approach to coordinating multi-robot manipulation through learned latent action spaces that are shared across different agents. We validate our method in simulated multi-robot manipulation tasks and demonstrate improvement over previous baselines in terms of sample efficiency and learning performance.
  arXiv  Detail & Related papers  (2022-11-28T23:20:47Z)
• Learning Multi-Arm Manipulation Through Collaborative Teleoperation [63.35924708783826]
  Imitation Learning (IL) is a powerful paradigm to teach robots to perform manipulation tasks. Many real-world tasks require multiple arms, such as lifting a heavy object or assembling a desk. We present Multi-Arm RoboTurk (MART), a multi-user data collection platform that allows multiple remote users to simultaneously teleoperate a set of robotic arms.
  arXiv  Detail & Related papers  (2020-12-12T05:43:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.