Bridging Handheld and Teleoperated Supervision for Contact-Rich Manipulation via State-Gated Experts

• URL: http://arxiv.org/abs/2606.26603v1
• Date: Thu, 25 Jun 2026 05:06:38 GMT
• Title: Bridging Handheld and Teleoperated Supervision for Contact-Rich Manipulation via State-Gated Experts
• Abstract summary: We study the interaction between handheld and teleoperated supervision for contact-rich manipulation.<n>We propose Bi-modal Routing for Imitation Data via Gated Experts (BRIDGE) to address the mismatch between observed and desired supervision.
• Score: 6.058807042621957
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Handheld data collection systems, such as the Universal Manipulation Interface (UMI), enable scalable data collection across diverse environments but only capture observed actions rather than the desired actions executed by a robot controller. In contrast, teleoperation captures desired actions directly, but is prohibitively time-consuming to collect. We revisit this trade-off through the lens of action validity across task phases. We observe that handheld trajectories provide valid supervision in tolerant, free-space phases, but lack dynamic feasibility in contact-sensitive phases, where tracking observed trajectories at high stiffness produces large, unsafe contact forces. We study the interaction between these two supervision types for contact-rich manipulation and find that training policies that combine handheld data with a small number of targeted teleoperated demonstrations provide an efficient hybrid strategy. Specifically, rather than teleoperating the entire task, we only collect partial teleoperated demonstrations for task segments where base handheld policies fail. However, naively mixing handheld and teleoperated phase-specific data yields worse performance than training on handheld data alone. To address this mismatch between observed and desired supervision, we propose Bi-modal Routing for Imitation Data via Gated Experts (BRIDGE), a mixture of diffusion policy experts that routes between specialist task phase heads conditioned on the current robot state. Notably, our approach enables task-phase specific use of desired actions during contact sensitive segments and improves success rates over handheld-only baselines by up to 36.7% across three contact-rich manipulation tasks.

Related papers

• Towards Human-level Dexterous Teleoperation [62.77371751776912]
  Human-level dexterous teleoperation requires precise control of object motion via dynamic hand-object contact.<n>We introduce TeleDexter, a hand-object co-tracking controller that maps operator intent into learned, low-level contact execution.<n>We evaluate TeleDexter on seven challenging dexterous teleoperation tasks spanning object reorientation and long-horizon tool use across two dexterous hands.
  arXiv  Detail & Related papers  (2026-07-13T12:36:47Z)
• HapTile: A Haptic-Informed Vision-Tactile-Language-Action Dataset for Contact-Rich Imitation Learning [24.64460457808404]
  HapTile is a contact-grounded visuotactile manipulation dataset that advances beyond vision-only trajectory datasets.<n>The data collection platform integrates haptic feedback directly into the teleoperation controller, enabling the operator to perceive contact interactions in real time.<n>The dataset comprises everyday manipulation tasks spanning a broad range of contact-rich skills, including pick-and-place, folding, pressing, stacking, and other routine activities.
  arXiv  Detail & Related papers  (2026-06-03T12:48:17Z)
• TAMEn: Tactile-Aware Manipulation Engine for Closed-Loop Data Collection in Contact-Rich Tasks [16.262130849751312]
  TAMEn is a tactile-aware manipulation engine for closed-loop data collection in contact-rich tasks.<n>Our system features a cross-morphology wearable interface that enables rapid adaptation across heterogeneous grippers.<n>Our proposed visuo-tactile learning framework increases task success rates from 34% to 75% across diverse bimanual manipulation tasks.
  arXiv  Detail & Related papers  (2026-04-08T17:49:35Z)
• TRIP-Bag: A Portable Teleoperation System for Plug-and-Play Robotic Arms and Leaders [7.528471207038859]
  TRIP-Bag is a portable, puppeteer-style teleoperation system fully contained within a commercial suitcase.<n>With a setup time of under five minutes and direct joint-to-joint teleoperation, TRIP-Bag enables rapid and reliable data collection in any environment.
  arXiv  Detail & Related papers  (2026-03-10T05:49:47Z)
• TouchGuide: Inference-Time Steering of Visuomotor Policies via Touch Guidance [53.35296919674763]
  TouchGuide is a novel cross-policy visuo-tactile fusion paradigm that fuses modalities within a low-dimensional action space.<n>touchGuide operates in two stages to guide a pre-trained diffusion or flow-matching visuomotor policy.<n>To facilitate TouchGuide training with high-quality and cost-effective data, we introduce TacUMI, a data collection system.
  arXiv  Detail & Related papers  (2026-01-28T04:22:47Z)
• TacUMI: A Multi-Modal Universal Manipulation Interface for Contact-Rich Tasks [35.05859151174601]
  We build on the idea of the handheld demonstration device Universal Manipulation Interface (UMI)<n>We introduce TacUMI, a multi-modal data collection system that integrates ViTac sensors, force-torque sensor, and pose tracker into a compact, robot-compatible gripper design.<n>We then propose a multi-modal segmentation framework that leverages temporal models to detect semantically meaningful event boundaries.
  arXiv  Detail & Related papers  (2026-01-21T00:14:28Z)
• Dexplore: Scalable Neural Control for Dexterous Manipulation from Reference-Scoped Exploration [58.4036440289082]
  Hand-object motion-capture (MoCap) offer large-scale, contact-rich demonstrations and hold promise for dexterous robotic scopes.<n>We introduce Dexplore, a unified single-loop optimization that performs repositories and tracking to learn robot control policies directly from MoCap at scale.
  arXiv  Detail & Related papers  (2025-09-11T17:59:07Z)
• The Role of Embodiment in Intuitive Whole-Body Teleoperation for Mobile Manipulation [20.65893345441958]
  A strong sense of embodiment combined with minimal physical and cognitive demands helps maintain data quality over extended periods.<n>We evaluate two visual feedback mechanisms: immersive virtual reality and conventional screen-based visualization of the robot's field of view.<n>Our results show that the use of VR as a feedback modality increases task completion time, cognitive workload, and perceived effort of the teleoperator.
  arXiv  Detail & Related papers  (2025-09-03T11:25:36Z)
• AnyPos: Automated Task-Agnostic Actions for Bimanual Manipulation [24.199522837278128]
  We present a new notion of task-agnostic action paradigm that decouples action execution from task-specific conditioning.<n>ATARA is a scalable self-supervised framework that accelerates collection by over $ 30times $ compared to human teleoperation.<n>We propose AnyPos, an inverse dynamics model equipped with Arm-Decoupled Estimation and a Direction-Aware Decoder.
  arXiv  Detail & Related papers  (2025-07-17T03:48:57Z)
• Gaze-guided Hand-Object Interaction Synthesis: Dataset and Method [61.19028558470065]
  We present GazeHOI, the first dataset to capture simultaneous 3D modeling of gaze, hand, and object interactions.<n>To tackle these issues, we propose a stacked gaze-guided hand-object interaction diffusion model, named GHO-Diffusion.<n>We also introduce HOI-Manifold Guidance during the sampling stage of GHO-Diffusion, enabling fine-grained control over generated motions.
  arXiv  Detail & Related papers  (2024-03-24T14:24:13Z)
• Error-Aware Imitation Learning from Teleoperation Data for Mobile Manipulation [54.31414116478024]
  In mobile manipulation (MM), robots can both navigate within and interact with their environment. In this work, we explore how to apply imitation learning (IL) to learn continuous visuo-motor policies for MM tasks.
  arXiv  Detail & Related papers  (2021-12-09T23:54:59Z)
• Human-in-the-Loop Imitation Learning using Remote Teleoperation [72.2847988686463]
  We build a data collection system tailored to 6-DoF manipulation settings. We develop an algorithm to train the policy iteratively on new data collected by the system. We demonstrate that agents trained on data collected by our intervention-based system and algorithm outperform agents trained on an equivalent number of samples collected by non-interventional demonstrators.
  arXiv  Detail & Related papers  (2020-12-12T05:30:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.