Prescient teleoperation of humanoid robots

• URL: http://arxiv.org/abs/2107.01281v1
• Date: Fri, 2 Jul 2021 21:10:35 GMT
• Title: Prescient teleoperation of humanoid robots
• Abstract summary: We introduce a system in which a humanoid robot executes commands before it actually receives them. The robot continuously predicts future commands by querying a machine learning model. An operator was able to successfully control a humanoid robot with delays up to 2 seconds in several whole-body manipulation tasks.
• Score: 6.230751621285322
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Humanoid robots could be versatile and intuitive human avatars that operate remotely in inaccessible places: the robot could reproduce in the remote location the movements of an operator equipped with a wearable motion capture device while sending visual feedback to the operator. While substantial progress has been made on transferring ("retargeting") human motions to humanoid robots, a major problem preventing the deployment of such systems in real applications is the presence of communication delays between the human input and the feedback from the robot: even a few hundred milliseconds of delay can irreversibly disturb the operator, let alone a few seconds. To overcome these delays, we introduce a system in which a humanoid robot executes commands before it actually receives them, so that the visual feedback appears to be synchronized to the operator, whereas the robot executed the commands in the past. To do so, the robot continuously predicts future commands by querying a machine learning model that is trained on past trajectories and conditioned on the last received commands. In our experiments, an operator was able to successfully control a humanoid robot (32 degrees of freedom) with stochastic delays up to 2 seconds in several whole-body manipulation tasks, including reaching different targets, picking up, and placing a box at distinct locations.

Related papers

• AHEAD: Anticipatory Hand-Driven Teleoperation via Human Intent Prediction [15.569640550342674]
  AHEAD is a real-time VR teleoperation system that anticipates operator intent to enable proactive, hand-driven control.<n>AHEAD reduces robot reaction latency by 0.6 s (object) and 1.4 s (slot) relative to baselines.
  arXiv  Detail & Related papers  (2026-07-16T16:21:40Z)
• HumanEgo: Zero-Shot Robot Learning from Minutes of Human Egocentric Videos [58.9564236347451]
  HumanEgo is a framework that bridges the embodiment gap between human and robot.<n>It lifts each human demonstration to an entity-level representation of hand-object interaction.<n>HumanEgo is robot-data-free, hardware-agnostic, data-efficient, and zero-shot human-to-robot transferable.
  arXiv  Detail & Related papers  (2026-05-24T08:26:41Z)
• ZeroWBC: Learning Natural Visuomotor Humanoid Control Directly from Human Egocentric Video [52.78703020909145]
  We introduce ZeroWBC, a novel framework that learns a natural humanoid visuomotor control policy directly from human egocentric videos.<n>Our approach first fine-tunes a Vision-Language Model (VLM) to predict future whole-body human motions based on text instructions and egocentric visual context.<n>Experiments on the Unitree G1 humanoid robot demonstrate that our method outperforms baseline approaches in motion naturalness and versatility.
  arXiv  Detail & Related papers  (2026-03-10T04:19:43Z)
• Inferring Operator Emotions from a Motion-Controlled Robotic Arm [12.761185664174759]
  A remote robot operator's affective state can significantly impact the resulting robot's motions.<n>Current emotion recognition methods rely on reading the user's vital signs or body language.<n>We demonstrate that the functional movements of a remote-controlled robotic avatar can be used to infer the emotional state of the human operator.
  arXiv  Detail & Related papers  (2025-12-09T19:57:43Z)
• Design and Control of a Bipedal Robotic Character [3.650193138379926]
  This work aims to unify expressive, artist-directed motions and robust dynamic mobility for legged robots. We introduce a new bipedal robot, designed with a focus on character-driven mechanical features. We present a reinforcement learning-based control architecture to robustly execute artistic motions conditioned on command signals.
  arXiv  Detail & Related papers  (2025-01-09T12:55:21Z)
• Generalizable Humanoid Manipulation with 3D Diffusion Policies [66.78220965526732]
  We build a real-world robotic system to address the problem of autonomous manipulation by humanoid robots.<n>Our system is mainly an integration of 1) a whole-upper-body robotic teleoperation system to acquire human-like robot data, and 2) a 25-DoF humanoid robot platform with a height-adjustable cart and a 3D LiDAR sensor.<n>We show that using only data collected in one scene and with only onboard computing, a full-sized humanoid robot can autonomously perform skills in diverse real-world scenarios.
  arXiv  Detail & Related papers  (2024-10-14T17:59:00Z)
• Unifying 3D Representation and Control of Diverse Robots with a Single Camera [48.279199537720714]
  We introduce Neural Jacobian Fields, an architecture that autonomously learns to model and control robots from vision alone. Our approach achieves accurate closed-loop control and recovers the causal dynamic structure of each robot.
  arXiv  Detail & Related papers  (2024-07-11T17:55:49Z)
• ImitationNet: Unsupervised Human-to-Robot Motion Retargeting via Shared Latent Space [9.806227900768926]
  This paper introduces a novel deep-learning approach for human-to-robot motion. Our method does not require paired human-to-robot data, which facilitates its translation to new robots. Our model outperforms existing works regarding human-to-robot similarity in terms of efficiency and precision.
  arXiv  Detail & Related papers  (2023-09-11T08:55:04Z)
• Giving Robots a Hand: Learning Generalizable Manipulation with Eye-in-Hand Human Video Demonstrations [66.47064743686953]
  Eye-in-hand cameras have shown promise in enabling greater sample efficiency and generalization in vision-based robotic manipulation. Videos of humans performing tasks, on the other hand, are much cheaper to collect since they eliminate the need for expertise in robotic teleoperation. In this work, we augment narrow robotic imitation datasets with broad unlabeled human video demonstrations to greatly enhance the generalization of eye-in-hand visuomotor policies.
  arXiv  Detail & Related papers  (2023-07-12T07:04:53Z)
• Zero-Shot Robot Manipulation from Passive Human Videos [59.193076151832145]
  We develop a framework for extracting agent-agnostic action representations from human videos. Our framework is based on predicting plausible human hand trajectories. We deploy the trained model zero-shot for physical robot manipulation tasks.
  arXiv  Detail & Related papers  (2023-02-03T21:39:52Z)
• HERD: Continuous Human-to-Robot Evolution for Learning from Human Demonstration [57.045140028275036]
  We show that manipulation skills can be transferred from a human to a robot through the use of micro-evolutionary reinforcement learning. We propose an algorithm for multi-dimensional evolution path searching that allows joint optimization of both the robot evolution path and the policy.
  arXiv  Detail & Related papers  (2022-12-08T15:56:13Z)
• Robots with Different Embodiments Can Express and Influence Carefulness in Object Manipulation [104.5440430194206]
  This work investigates the perception of object manipulations performed with a communicative intent by two robots. We designed the robots' movements to communicate carefulness or not during the transportation of objects.
  arXiv  Detail & Related papers  (2022-08-03T13:26:52Z)
• Robotic Telekinesis: Learning a Robotic Hand Imitator by Watching Humans on Youtube [24.530131506065164]
  We build a system that enables any human to control a robot hand and arm, simply by demonstrating motions with their own hand. The robot observes the human operator via a single RGB camera and imitates their actions in real-time. We leverage this data to train a system that understands human hands and retargets a human video stream into a robot hand-arm trajectory that is smooth, swift, safe, and semantically similar to the guiding demonstration.
  arXiv  Detail & Related papers  (2022-02-21T18:59:59Z)
• Know Thyself: Transferable Visuomotor Control Through Robot-Awareness [22.405839096833937]
  Training visuomotor robot controllers from scratch on a new robot typically requires generating large amounts of robot-specific data. We propose a "robot-aware" solution paradigm that exploits readily available robot "self-knowledge" Our experiments on tabletop manipulation tasks in simulation and on real robots demonstrate that these plug-in improvements dramatically boost the transferability of visuomotor controllers.
  arXiv  Detail & Related papers  (2021-07-19T17:56:04Z)
• Target Reaching Behaviour for Unfreezing the Robot in a Semi-Static and Crowded Environment [2.055949720959582]
  We propose a robot behavior for a wheeled humanoid robot that complains with social norms for clearing its path when the robot is frozen due to the presence of humans. The behavior consists of two modules: 1) A detection module, which make use of the Yolo v3 algorithm trained to detect human hands and human arms, and 2) A gesture module, which make use of a policy trained in simulation using the Proximal Policy Optimization algorithm.
  arXiv  Detail & Related papers  (2020-12-02T13:43:59Z)
• Morphology-Agnostic Visual Robotic Control [76.44045983428701]
  MAVRIC is an approach that works with minimal prior knowledge of the robot's morphology. We demonstrate our method on visually-guided 3D point reaching, trajectory following, and robot-to-robot imitation.
  arXiv  Detail & Related papers  (2019-12-31T15:45:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.