A Digital Twin Framework for Virtual Visuo-Haptic Teleoperation of Complex-Shaped Optical Microrobots

• URL: http://arxiv.org/abs/2605.28448v1
• Date: Wed, 27 May 2026 13:15:38 GMT
• Title: A Digital Twin Framework for Virtual Visuo-Haptic Teleoperation of Complex-Shaped Optical Microrobots
• Abstract summary: This paper presents a digital twin framework for virtual visuo-haptic teleoperation of complex-shaped optical microrobots.<n>The framework integrates a digital twin environment, image-based pose and depth estimation, microrobot motion simulation, and model-based haptic rendering.<n>Results show the framework's effectiveness for evaluating visuo-haptic teleoperation strategies for complex-shaped optical microrobots.
• Score: 6.975054201075641
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Optical tweezers (OT) provide piconewton-scale manipulation for delicate biomedical tasks, where visuo-haptic feedback can improve operator awareness by conveying interaction-force cues and trap-stability information. However, visuo-haptic teleoperation frameworks for complex-shaped optical microrobots remain underdeveloped, particularly in multi-trap manipulation scenarios. This paper presents a digital twin framework for virtual visuo-haptic teleoperation of complex-shaped OT-driven microrobots. The framework integrates a digital twin environment, image-based pose and depth estimation, microrobot motion simulation, and model-based haptic rendering within a Robot Operating System (ROS)-connected bimanual teleoperation system. For force modeling, we combine a Multi-Sphere Distributed Manipulation (MSDM) model with optical-force estimation from the Optical Tweezers Toolbox, enabling simulator-driven visuo-haptic feedback. The framework reproduces representative microrobot motion trends and provides haptic force rendering that is numerically consistent with the fitted optical-force model. In simulated cell-delivery tasks, haptic feedback reduced the standard deviations of the contact-force metric and the microrobot-to-trap-center distance metric by 53.2% and 55.2%, respectively, and improved task success from 30% to 80%. These results demonstrate the framework's effectiveness for evaluating visuo-haptic teleoperation strategies for complex-shaped optical microrobots.

Related papers

• Real-Time sEMG-Based Telecontrol of an Assistive Robotic Arm Using a 1D Convolutional Neural Network [0.30786914102688595]
  This work proposes a real-time sEMG-based interface for the teleoperation of an assistive robotic arm.<n>The system relies on four-channel sEMG acquisition, signal preprocessing, segmentation into sliding windows, and classification using a one-dimensional convolutional neural network (CNN)<n>The system exhibits stable real-time behavior, with an average latency of approximately 0.32 s consistent with the chosen windowing strategy.
  arXiv  Detail & Related papers  (2026-07-14T15:34:28Z)
• Dual-Control Frequency-Aware Diffusion Model for Depth-Dependent Optical Microrobot Microscopy Image Generation [7.32096324372237]
  We propose Du-FreqNet, a dual-control, frequency-aware diffusion model for physically consistent microscopy image synthesis.<n>By leveraging differentiable FFT-based supervision, Du-FreqNet captures physically meaningful frequency distributions often missed by pixel-space methods.<n>Our model achieves controllable, depth-dependent image synthesis, improving SSIM by 20.7% over baselines.
  arXiv  Detail & Related papers  (2026-04-13T16:24:08Z)
• Robotic Grasping and Placement Controlled by EEG-Based Hybrid Visual and Motor Imagery [64.82869118243723]
  We present a framework that integrates EEG-based visual and motor imagery (VI/MI) with robotic control to enable real-time, intention-driven grasping and placement.<n>Motivated by the promise of BCI-driven robotics, this system bridges neural signals with physical control by deploying offline-pretrained decoders in a zero-shot manner.
  arXiv  Detail & Related papers  (2026-03-03T17:41:42Z)
• Real-time Monocular 2D and 3D Perception of Endoluminal Scenes for Controlling Flexible Robotic Endoscopic Instruments [28.77639956401214]
  This paper presents a visual perception platform for a continuum robotic system in endoluminal surgery.<n>We introduce 2D and 3D learning-based perception algorithms to identify position and orientation of flexible instruments.<n>A physically-realistic simulator generates realistic endoluminal scenes, enabling control of flexible robots.
  arXiv  Detail & Related papers  (2026-02-16T11:46:14Z)
• MiDAS: A Multimodal Data Acquisition System and Dataset for Robot-Assisted Minimally Invasive Surgery [4.287077788799387]
  MiDAS is a platform-agnostic system for time-synchronized, non-invasive multimodal data acquisition across surgical robotic platforms.<n>We validated MiDAS on the open-source Raven-II and the clinical da Vinci Xi by collecting multimodal datasets of peg transfer and hernia repair suturing tasks.
  arXiv  Detail & Related papers  (2026-02-12T20:56:15Z)
• Physics-Informed Machine Learning for Efficient Sim-to-Real Data Augmentation in Micro-Object Pose Estimation [6.975054201075641]
  This work proposes a novel physics-informed deep generative learning framework for pose estimation of microrobots.<n>It integrates wave optics-based physical rendering and depth alignment into a generative adversarial network (GAN)<n>Our method improves the structural similarity index (SSIM) by 35.6% compared to purely AI-driven methods.<n>It achieves 93.9%/91.9% (pitch/roll) accuracy, just 5.0%/5.4% (pitch/roll) below that of an estimator trained exclusively on real data.
  arXiv  Detail & Related papers  (2025-11-20T16:10:53Z)
• Accelerating 3D Photoacoustic Computed Tomography with End-to-End Physics-Aware Neural Operators [74.65171736966131]
  Photoacoustic computed tomography (PACT) combines optical contrast with ultrasonic resolution, achieving deep-tissue imaging beyond the optical diffusion limit.<n>Current implementations require dense transducer arrays and prolonged acquisition times, limiting clinical translation.<n>We introduce Pano, an end-to-end physics-aware model that directly learns the inverse acoustic mapping from sensor measurements to volumetric reconstructions.
  arXiv  Detail & Related papers  (2025-09-11T23:12:55Z)
• Is Single-View Mesh Reconstruction Ready for Robotics? [78.14584238127338]
  We evaluate single-view mesh reconstruction models for their potential in enabling instant digital twin creation for real-time planning and dynamics prediction using physics simulators for robotic manipulation.<n>Our findings highlight critical gaps between computer vision advances and robotics needs, guiding future research at this intersection.
  arXiv  Detail & Related papers  (2025-05-23T14:35:56Z)
• Taccel: Scaling Up Vision-based Tactile Robotics via High-performance GPU Simulation [34.47272224723296]
  We present Taccel, a high-performance simulation platform that integrates IPC and ABD to model robots, tactile sensors, and objects with both accuracy and unprecedented speed.<n>Unlike previous simulators that operate at sub-real-time speeds with limited parallelization, Taccel provides precise physics simulation and realistic tactile signals.<n>These capabilities position Taccel as a powerful tool for scaling up tactile robotics research and development, potentially transforming how robots interact with and understand their physical environment.
  arXiv  Detail & Related papers  (2025-04-17T12:57:11Z)
• Image-to-Force Estimation for Soft Tissue Interaction in Robotic-Assisted Surgery Using Structured Light [4.199824399433837]
  This letter introduces an effective vision-based scheme that utilizes a One-Shot structured light projection with a designed pattern on soft tissue.<n>The images captured from the endoscopic stereo camera are analyzed to reconstruct high-resolution 3D point clouds for soft tissue deformation.<n>A modified PointNet-based force estimation method is proposed, which excels in representing the complex mechanical properties of soft tissue.
  arXiv  Detail & Related papers  (2025-01-15T05:36:41Z)
• MS-MANO: Enabling Hand Pose Tracking with Biomechanical Constraints [50.61346764110482]
  We integrate a musculoskeletal system with a learnable parametric hand model, MANO, to create MS-MANO. This model emulates the dynamics of muscles and tendons to drive the skeletal system, imposing physiologically realistic constraints on the resulting torque trajectories. We also propose a simulation-in-the-loop pose refinement framework, BioPR, that refines the initial estimated pose through a multi-layer perceptron network.
  arXiv  Detail & Related papers  (2024-04-16T02:18:18Z)
• Robotic Navigation Autonomy for Subretinal Injection via Intelligent Real-Time Virtual iOCT Volume Slicing [88.99939660183881]
  We propose a framework for autonomous robotic navigation for subretinal injection. Our method consists of an instrument pose estimation method, an online registration between the robotic and the i OCT system, and trajectory planning tailored for navigation to an injection target. Our experiments on ex-vivo porcine eyes demonstrate the precision and repeatability of the method.
  arXiv  Detail & Related papers  (2023-01-17T21:41:21Z)
• Elastic Tactile Simulation Towards Tactile-Visual Perception [58.44106915440858]
  We propose Elastic Interaction of Particles (EIP) for tactile simulation. EIP models the tactile sensor as a group of coordinated particles, and the elastic property is applied to regulate the deformation of particles during contact. We further propose a tactile-visual perception network that enables information fusion between tactile data and visual images.
  arXiv  Detail & Related papers  (2021-08-11T03:49:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.