Uncertainty-Aware Intention Prediction for Human-to-Robot Assembly Teleoperation

• URL: http://arxiv.org/abs/2606.08341v1
• Date: Sat, 06 Jun 2026 21:13:57 GMT
• Title: Uncertainty-Aware Intention Prediction for Human-to-Robot Assembly Teleoperation
• Abstract summary: In assisted teleoperation for human-robot collaboration, accurate intention prediction is critical for enabling timely and reliable robotic assistance.<n>We propose an uncertainty-aware human-to-robot intention prediction framework that combines hierarchical transfer learning, frame-level prediction sets, and VLM-guided segment correction.<n>Experiments show that human-to-robot fine-tuning improves the robot test-set Edit score from 70.50 to 80.70 using only 16 robot demonstrations.
• Score: 8.460582272894202
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In assisted teleoperation for human-robot collaboration, accurate intention prediction is critical for enabling timely and reliable robotic assistance during long-horizon manipulation and assembly tasks. These systems require continuous understanding of user behavior to recognize actions, anticipate intentions, and detect mistakes in real time. However, robot teleoperation demonstrations are costly and hardware-limited, whereas human demonstrations are easier to collect and provide rich temporal structure. To address this challenge, we propose an uncertainty-aware human-to-robot intention prediction framework that combines: (1) hierarchical transfer learning, where MS-TCN++ is pretrained on human hand demonstrations and fine-tuned on limited robot teleoperation data to capture low-level actions and high-level task intentions; (2) a conformal prediction module that provides frame-level prediction sets with statistical coverage guarantees for reliable uncertainty quantification and early intention estimation; and (3) VLM-guided segment correction, which selectively reviews low-confidence or temporally uncertain segments using visual and temporal context. The framework supports action recognition, temporal segmentation, intention anticipation, and mistake detection for assisted teleoperation. Experiments on robot assembly demonstrations with 22 action classes show that human-to-robot fine-tuning improves the robot test-set Edit score from 70.50 to 80.70 using only 16 robot demonstrations. Edit-safe VLM correction further improves frame accuracy from 45.21% to 46.42% and increases F1@25 and F1@50 while preserving the Edit score. These results show that human demonstrations provide scalable pretraining data for robust, uncertainty-aware robot action segmentation. Code and data: project website.

Related papers

• RGB-D Video Generation for Improving Human-to-Robot Object Handover Prediction [51.76737991200891]
  Human-to-robot (H2R) object handover is a fundamental capability for human-robot collaboration, yet progress is hindered by the scarcity of large-scale, human-centric datasets.<n>We introduce Hand2Bot, an RGB-D video dataset that provides rich contextual information such as body posture and facial expressions.<n>We propose PassGen, a generative pipeline that synthesizes realistic handover sequences while ensuring hand-object consistency.
  arXiv  Detail & Related papers  (2026-08-13T09:55:59Z)
• AHEAD: Anticipatory Hand-Driven Teleoperation via Human Intent Prediction [15.569640550342674]
  AHEAD is a real-time VR teleoperation system that anticipates operator intent to enable proactive, hand-driven control.<n>AHEAD reduces robot reaction latency by 0.6 s (object) and 1.4 s (slot) relative to baselines.
  arXiv  Detail & Related papers  (2026-07-16T16:21:40Z)
• Closing the Loop in Teleoperation: Episode-Level Data Quality Assessment and Feedback for High-Quality Demonstration Collection [3.424028268271705]
  We present a framework that closes the loop in teleoperation by providing immediate post-episode feedback grounded in semantic task progress and robot telemetry.<n>Unlike binary success or failure feedback, the proposed system explains why an episode is suboptimal and highlights specific behaviors to correct in the next trial.
  arXiv  Detail & Related papers  (2026-05-25T21:52:08Z)
• Robotic Grasping and Placement Controlled by EEG-Based Hybrid Visual and Motor Imagery [64.82869118243723]
  We present a framework that integrates EEG-based visual and motor imagery (VI/MI) with robotic control to enable real-time, intention-driven grasping and placement.<n>Motivated by the promise of BCI-driven robotics, this system bridges neural signals with physical control by deploying offline-pretrained decoders in a zero-shot manner.
  arXiv  Detail & Related papers  (2026-03-03T17:41:42Z)
• HHI-Assist: A Dataset and Benchmark of Human-Human Interaction in Physical Assistance Scenario [63.77482302352545]
  HHI-Assist is a dataset comprising motion capture clips of human-human interactions in assistive tasks.<n>Our work has the potential to significantly enhance robotic assistance policies.
  arXiv  Detail & Related papers  (2025-09-12T09:38:17Z)
• Physical Autoregressive Model for Robotic Manipulation without Action Pretraining [65.8971623698511]
  We build upon autoregressive video generation models to propose a Physical Autoregressive Model (PAR)<n>PAR leverages the world knowledge embedded in video pretraining to understand physical dynamics without requiring action pretraining.<n>Experiments on the ManiSkill benchmark show that PAR achieves a 100% success rate on the PushCube task.
  arXiv  Detail & Related papers  (2025-08-13T13:54:51Z)
• Conformalized Teleoperation: Confidently Mapping Human Inputs to High-Dimensional Robot Actions [4.855534476454559]
  We learn a mapping from low-dimensional human inputs to high-dimensional robot actions. Our key idea is to adapt the assistive map at training time to additionally estimate high-dimensional action quantiles. We propose an uncertainty-interval-based mechanism for detecting high-uncertainty user inputs and robot states.
  arXiv  Detail & Related papers  (2024-06-11T23:16:46Z)
• Model-Based Runtime Monitoring with Interactive Imitation Learning [30.70994322652745]
  This work aims to endow a robot with the ability to monitor and detect errors during task execution. We introduce a model-based runtime monitoring algorithm that learns from deployment data to detect system anomalies and anticipate failures. Our method outperforms the baselines across system-level and unit-test metrics, with 23% and 40% higher success rates in simulation and on physical hardware.
  arXiv  Detail & Related papers  (2023-10-26T16:45:44Z)
• ManiCast: Collaborative Manipulation with Cost-Aware Human Forecasting [8.274511768083665]
  We present ManiCast, a novel framework that learns cost-aware human forecasts and feeds them to a model predictive control planner. Our framework enables fluid, real-time interactions between a human and a 7-DoF robot arm across a number of real-world tasks.
  arXiv  Detail & Related papers  (2023-10-20T03:34:31Z)
• Self-Improving Robots: End-to-End Autonomous Visuomotor Reinforcement Learning [54.636562516974884]
  In imitation and reinforcement learning, the cost of human supervision limits the amount of data that robots can be trained on. In this work, we propose MEDAL++, a novel design for self-improving robotic systems. The robot autonomously practices the task by learning to both do and undo the task, simultaneously inferring the reward function from the demonstrations.
  arXiv  Detail & Related papers  (2023-03-02T18:51:38Z)
• Revisiting the Adversarial Robustness-Accuracy Tradeoff in Robot Learning [121.9708998627352]
  Recent work has shown that, in practical robot learning applications, the effects of adversarial training do not pose a fair trade-off. This work revisits the robustness-accuracy trade-off in robot learning by analyzing if recent advances in robust training methods and theory can make adversarial training suitable for real-world robot applications.
  arXiv  Detail & Related papers  (2022-04-15T08:12:15Z)
• Where is my hand? Deep hand segmentation for visual self-recognition in humanoid robots [129.46920552019247]
  We propose the use of a Convolution Neural Network (CNN) to segment the robot hand from an image in an egocentric view. We fine-tuned the Mask-RCNN network for the specific task of segmenting the hand of the humanoid robot Vizzy.
  arXiv  Detail & Related papers  (2021-02-09T10:34:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.