Observer Design for Augmented Reality-based Teleoperation of Soft Robots

• URL: http://arxiv.org/abs/2603.05015v1
• Date: Thu, 05 Mar 2026 10:04:14 GMT
• Title: Observer Design for Augmented Reality-based Teleoperation of Soft Robots
• Abstract summary: This paper presents an augmented reality interface for teleoperating soft robots.<n>The developed system consists of Microsoft HoloLens 2 glasses and a central computer responsible for calculations.<n>Using data collected from sensors, the computer estimates the robot's position based on the physics of the virtual reality programme.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Although virtual and augmented reality are gaining traction as teleoperation tools for various types of robots, including manipulators and mobile robots, they are not being used for soft robots. The inherent difficulties of modelling soft robots mean that combining accurate and computationally efficient representations is very challenging. This paper presents an augmented reality interface for teleoperating these devices. The developed system consists of Microsoft HoloLens 2 glasses and a central computer responsible for calculations. Validation is performed on PETER, a highly modular pneumatic manipulator. Using data collected from sensors, the computer estimates the robot's position based on the physics of the virtual reality programme. Errors obtained are on the order of 5% of the robot's length, demonstrating that augmented reality facilitates operator interaction with soft manipulators and can be integrated into the control loop.

Related papers

• ORCESTRA: VLM-driven Visual Robot programming in Mixed Reality [2.1645011609137295]
  ORCESTRA is a mixed-reality system for programming robot digital twins through no-code waypoint teaching and language-guided control.<n>In a passthrough mixed-reality workspace, users place robot twins on real surfaces, teach trajectories, save robot-relative episodes, or issue spoken/typed commands that a vision-language model converts into structured digital-twin plans.
  arXiv  Detail & Related papers  (2026-08-01T17:12:01Z)
• RynnWorld-Teleop: An Action-Conditioned World Model for Digital Teleoperation [48.828787848144806]
  We introduce digital teleoperation, a paradigm that decouples data collection from physical constraints by replacing real robot with a generative world model.<n>We instantiate this paradigm in RynnWorld-Teleop, a system that integrates depth-aware skeletal conditioning, progressive human-to-robot training on a video Diffusion Transformer, and streaming autoregressive distillation.<n>This pipeline compresses the generative process into a single-pass inference, enabling 40+ FPS, real-time interactive generation on a single H100 GPU.
  arXiv  Detail & Related papers  (2026-07-07T17:58:11Z)
• RobotSeg: A Model and Dataset for Segmenting Robots in Image and Video [56.9581053843815]
  We introduce RobotSeg, a foundation model for robot segmentation in image and video.<n>It addresses the lack of adaptation to articulated robots, reliance on manual prompts, and the need for per-frame training mask annotations.<n>It achieves state-of-the-art performance on both images and videos.
  arXiv  Detail & Related papers  (2025-11-28T07:51:02Z)
• Taccel: Scaling Up Vision-based Tactile Robotics via High-performance GPU Simulation [34.47272224723296]
  We present Taccel, a high-performance simulation platform that integrates IPC and ABD to model robots, tactile sensors, and objects with both accuracy and unprecedented speed.<n>Unlike previous simulators that operate at sub-real-time speeds with limited parallelization, Taccel provides precise physics simulation and realistic tactile signals.<n>These capabilities position Taccel as a powerful tool for scaling up tactile robotics research and development, potentially transforming how robots interact with and understand their physical environment.
  arXiv  Detail & Related papers  (2025-04-17T12:57:11Z)
• Physical Simulation for Multi-agent Multi-machine Tending [11.017120167486448]
  Reinforcement learning (RL) offers a promising solution where robots can learn through interaction with the environment. We leveraged a simplistic robotic system to work with RL with "real" data without having to deploy large expensive robots in a manufacturing setting.
  arXiv  Detail & Related papers  (2024-10-11T17:57:44Z)
• Controlling diverse robots by inferring Jacobian fields with deep networks [48.279199537720714]
  Mirroring the complex structures and diverse functions of natural organisms is a long-standing challenge in robotics.<n>We introduce a method that uses deep neural networks to map a video stream of a robot to its visuomotor Jacobian field.<n>Our approach achieves accurate closed-loop control and recovers the causal dynamic structure of each robot.
  arXiv  Detail & Related papers  (2024-07-11T17:55:49Z)
• RoboScript: Code Generation for Free-Form Manipulation Tasks across Real and Simulation [77.41969287400977]
  This paper presents textbfRobotScript, a platform for a deployable robot manipulation pipeline powered by code generation. We also present a benchmark for a code generation benchmark for robot manipulation tasks in free-form natural language. We demonstrate the adaptability of our code generation framework across multiple robot embodiments, including the Franka and UR5 robot arms.
  arXiv  Detail & Related papers  (2024-02-22T15:12:00Z)
• Giving Robots a Hand: Learning Generalizable Manipulation with Eye-in-Hand Human Video Demonstrations [66.47064743686953]
  Eye-in-hand cameras have shown promise in enabling greater sample efficiency and generalization in vision-based robotic manipulation. Videos of humans performing tasks, on the other hand, are much cheaper to collect since they eliminate the need for expertise in robotic teleoperation. In this work, we augment narrow robotic imitation datasets with broad unlabeled human video demonstrations to greatly enhance the generalization of eye-in-hand visuomotor policies.
  arXiv  Detail & Related papers  (2023-07-12T07:04:53Z)
• AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation System [51.48191418148764]
  Vision-based teleoperation can endow robots with human-level intelligence to interact with the environment. Current vision-based teleoperation systems are designed and engineered towards a particular robot model and deploy environment. We propose AnyTeleop, a unified and general teleoperation system to support multiple different arms, hands, realities, and camera configurations within a single system.
  arXiv  Detail & Related papers  (2023-07-10T14:11:07Z)
• A Virtual Reality Teleoperation Interface for Industrial Robot Manipulators [10.331963200885774]
  We address the problem of teleoperating an industrial robot manipulator via a commercially available Virtual Reality interface. We find that applying standard practices for VR control of robot arms is challenging for industrial platforms. We propose a simplified filtering approach to process command signals to enable operators to effectively teleoperate industrial robot arms with VR interfaces.
  arXiv  Detail & Related papers  (2023-05-18T13:26:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.