A High-Fidelity Robotic Manipulator Teleoperation Framework for Human-Centered Augmented Reality Evaluation

• URL: http://arxiv.org/abs/2602.06273v1
• Date: Fri, 06 Feb 2026 00:21:06 GMT
• Title: A High-Fidelity Robotic Manipulator Teleoperation Framework for Human-Centered Augmented Reality Evaluation
• Abstract summary: ARBot is a real-time teleoperation platform that can effectively capture natural human motion and accurately replay the movements via robotic manipulators.<n>We release a benchmark dataset of 132 human and synthetic trajectories captured using ARBot to support controllable and scalable AR evaluation.
• Score: 1.8381910913370485
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Validating Augmented Reality (AR) tracking and interaction models requires precise, repeatable ground-truth motion. However, human users cannot reliably perform consistent motion due to biomechanical variability. Robotic manipulators are promising to act as human motion proxies if they can mimic human movements. In this work, we design and implement ARBot, a real-time teleoperation platform that can effectively capture natural human motion and accurately replay the movements via robotic manipulators. ARBot includes two capture models: stable wrist motion capture via a custom CV and IMU pipeline, and natural 6-DOF control via a mobile application. We design a proactively-safe QP controller to ensure smooth, jitter-free execution of the robotic manipulator, enabling it to function as a high-fidelity record and replay physical proxy. We open-source ARBot and release a benchmark dataset of 132 human and synthetic trajectories captured using ARBot to support controllable and scalable AR evaluation.

Related papers

• Real-Time Whole-Body Teleoperation of a Humanoid Robot Using IMU-Based Motion Capture with Sim2Sim and Sim2Real Validation [0.7325481380690131]
  This paper presents a real-time whole-body teleoperation system recorded with a Unitree G1 humanoid robot.<n>We introduce a custom motion-processing, kinematic, and control pipeline engineered for continuous, low-latency operation without any offline buffering or learning-based components.<n> Experimental results demonstrate stable, synchronized reproduction of a broad motion repertoire, including walking, standing, sitting, turning, bowing, and coordinated expressive full-body gestures.
  arXiv  Detail & Related papers  (2026-05-12T16:22:17Z)
• BifrostUMI: Bridging Robot-Free Demonstrations and Humanoid Whole-Body Manipulation [6.733928872257592]
  BifrostUMI is a portable, efficient, and robot-free data collection framework for humanoid robots.<n>We demonstrate the efficacy and versatility of the proposed framework across two distinct experimental scenarios.
  arXiv  Detail & Related papers  (2026-05-05T07:35:09Z)
• ZeroWBC: Learning Natural Visuomotor Humanoid Control Directly from Human Egocentric Video [52.78703020909145]
  We introduce ZeroWBC, a novel framework that learns a natural humanoid visuomotor control policy directly from human egocentric videos.<n>Our approach first fine-tunes a Vision-Language Model (VLM) to predict future whole-body human motions based on text instructions and egocentric visual context.<n>Experiments on the Unitree G1 humanoid robot demonstrate that our method outperforms baseline approaches in motion naturalness and versatility.
  arXiv  Detail & Related papers  (2026-03-10T04:19:43Z)
• Imitating What Works: Simulation-Filtered Modular Policy Learning from Human Videos [56.510263910611684]
  We tackle prehensile manipulation, in which tasks involve grasping an object before performing various post-grasp motions.<n>Human videos offer strong signals for learning the post-grasp motions, but they are less useful for learning the prerequisite grasping behaviors.<n>We present Perceive-Simulate-Imitate (PSI), a framework for training a modular manipulation policy using human video motion data.
  arXiv  Detail & Related papers  (2026-02-13T18:59:10Z)
• MiVLA: Towards Generalizable Vision-Language-Action Model with Human-Robot Mutual Imitation Pre-training [102.850162490626]
  We propose MiVLA, a vision-language-action model empowered by human-robot mutual imitation pre-training.<n>We show that MiVLA achieves strong improved generalization capability, outperforming state-of-the-art VLAs.
  arXiv  Detail & Related papers  (2025-12-17T12:59:41Z)
• CHIP: Adaptive Compliance for Humanoid Control through Hindsight Perturbation [70.5382178207975]
  hIsight Perturbation (CHIP) is a plug-and-play module that enables controllable end-effector stiffness.<n>CHIP is easy to implement and requires neither data augmentation nor additional reward tuning.<n>We show that a generalist motion-tracking controller trained with CHIP can perform a diverse set of forceful manipulation tasks.
  arXiv  Detail & Related papers  (2025-12-16T18:56:04Z)
• Implicit Kinodynamic Motion Retargeting for Human-to-humanoid Imitation Learning [35.8296790596745]
  Implicit Kinodynamic Motion Retargeting (IKMR) is a novel efficient and scalable framework that considers both kinematics and dynamics.<n>IKMR pretrains motion topology representation and a dual encoder-decoder architecture to learn a motion domain mapping.<n>We conduct our experiments both in the simulator and the real robot on a full-size humanoid robot.
  arXiv  Detail & Related papers  (2025-09-18T21:34:02Z)
• VidBot: Learning Generalizable 3D Actions from In-the-Wild 2D Human Videos for Zero-Shot Robotic Manipulation [53.63540587160549]
  VidBot is a framework enabling zero-shot robotic manipulation using learned 3D affordance from in-the-wild monocular RGB-only human videos.<n> VidBot paves the way for leveraging everyday human videos to make robot learning more scalable.
  arXiv  Detail & Related papers  (2025-03-10T10:04:58Z)
• Human-Agent Joint Learning for Efficient Robot Manipulation Skill Acquisition [48.65867987106428]
  We introduce a novel system for joint learning between human operators and robots. It enables human operators to share control of a robot end-effector with a learned assistive agent. It reduces the need for human adaptation while ensuring the collected data is of sufficient quality for downstream tasks.
  arXiv  Detail & Related papers  (2024-06-29T03:37:29Z)
• Hierarchical Procedural Framework for Low-latency Robot-Assisted Hand-Object Interaction [45.256762954338704]
  We propose a hierarchical procedural framework to enable robot-assisted hand-object interaction (HOI)<n>An open-loop hierarchy leverages the RGB-based 3D reconstruction of the human hand, based on which motion primitives have been designed to translate hand motions into robotic actions.<n>A case study of ring-wearing tasks indicates the potential application of this work in assistive technologies such as healthcare and manufacturing.
  arXiv  Detail & Related papers  (2024-05-29T21:20:16Z)
• Naturalistic Robot Arm Trajectory Generation via Representation Learning [4.7682079066346565]
  Integration of manipulator robots in household environments suggests a need for more predictable human-like robot motion. One method of generating naturalistic motion trajectories is via imitation of human demonstrators. This paper explores a self-supervised imitation learning method using an autoregressive neural network for an assistive drinking task.
  arXiv  Detail & Related papers  (2023-09-14T09:26:03Z)
• Human-Robot Skill Transfer with Enhanced Compliance via Dynamic Movement Primitives [1.7901837062462316]
  We introduce a systematic method to extract the dynamic features from human demonstration to auto-tune the parameters in the Dynamic Movement Primitives framework. Our method was implemented into an actual human-robot setup to extract human dynamic features and used to regenerate the robot trajectories following both LfD and RL.
  arXiv  Detail & Related papers  (2023-04-12T08:48:28Z)
• A Transferable Legged Mobile Manipulation Framework Based on Disturbance Predictive Control [15.044159090957292]
  Legged mobile manipulation, where a quadruped robot is equipped with a robotic arm, can greatly enhance the performance of the robot. We propose a unified framework disturbance predictive control where a reinforcement learning scheme with a latent dynamic adapter is embedded into our proposed low-level controller.
  arXiv  Detail & Related papers  (2022-03-02T14:54:10Z)
• Show Me What You Can Do: Capability Calibration on Reachable Workspace for Human-Robot Collaboration [83.4081612443128]
  We show that a short calibration using REMP can effectively bridge the gap between what a non-expert user thinks a robot can reach and the ground-truth. We show that this calibration procedure not only results in better user perception, but also promotes more efficient human-robot collaborations.
  arXiv  Detail & Related papers  (2021-03-06T09:14:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.