Setup-Invariant Augmented Reality for Teaching by Demonstration with Surgical Robots

• URL: http://arxiv.org/abs/2504.06677v1
• Date: Wed, 09 Apr 2025 08:34:25 GMT
• Title: Setup-Invariant Augmented Reality for Teaching by Demonstration with Surgical Robots
• Authors: Alexandre Banks, Richard Cook, Septimiu E. Salcudean,
• Abstract summary: We present dV-STEAR, an open-source system that plays back task-aligned expert demonstrations without assuming identical setup joint positions between expert and novice.<n>In a user study, dV-STEAR significantly improved novice performance on tasks from the Fundamentals of Laparoscopic Surgery.<n>Across both tasks, participants using dV-STEAR exhibited significantly more balanced hand use and reported lower frustration levels.
• Score: 49.26692555627371
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Augmented reality (AR) is an effective tool in robotic surgery education as it combines exploratory learning with three-dimensional guidance. However, existing AR systems require expert supervision and do not account for differences in the mentor and mentee robot configurations. To enable novices to train outside the operating room while receiving expert-informed guidance, we present dV-STEAR: an open-source system that plays back task-aligned expert demonstrations without assuming identical setup joint positions between expert and novice. Pose estimation was rigorously quantified, showing a registration error of 3.86 (SD=2.01)mm. In a user study (N=24), dV-STEAR significantly improved novice performance on tasks from the Fundamentals of Laparoscopic Surgery. In a single-handed ring-over-wire task, dV-STEAR increased completion speed (p=0.03) and reduced collision time (p=0.01) compared to dry-lab training alone. During a pick-and-place task, it improved success rates (p=0.004). Across both tasks, participants using dV-STEAR exhibited significantly more balanced hand use and reported lower frustration levels. This work presents a novel educational tool implemented on the da Vinci Research Kit, demonstrates its effectiveness in teaching novices, and builds the foundation for further AR integration into robot-assisted surgery.

Related papers

• dARt Vinci: Egocentric Data Collection for Surgical Robot Learning at Scale [9.732963094538837]
  We introduce dARt Vinci, a scalable data collection platform for robot learning in surgical settings.<n>The system uses Augmented Reality (AR) hand tracking and a high-fidelity physics engine to capture subtle maneuvers in primitive surgical tasks.
  arXiv  Detail & Related papers  (2025-03-07T18:07:54Z)
• Offline Imitation Learning Through Graph Search and Retrieval [57.57306578140857]
  Imitation learning is a powerful machine learning algorithm for a robot to acquire manipulation skills. We propose GSR, a simple yet effective algorithm that learns from suboptimal demonstrations through Graph Search and Retrieval. GSR can achieve a 10% to 30% higher success rate and over 30% higher proficiency compared to baselines.
  arXiv  Detail & Related papers  (2024-07-22T06:12:21Z)
• RILe: Reinforced Imitation Learning [60.63173816209543]
  RILe is a framework that combines the strengths of imitation learning and inverse reinforcement learning to learn a dense reward function efficiently.<n>Our framework produces high-performing policies in high-dimensional tasks where direct imitation fails to replicate complex behaviors.
  arXiv  Detail & Related papers  (2024-06-12T17:56:31Z)
• Demonstration-Guided Reinforcement Learning with Efficient Exploration for Task Automation of Surgical Robot [54.80144694888735]
  We introduce Demonstration-guided EXploration (DEX), an efficient reinforcement learning algorithm. Our method estimates expert-like behaviors with higher values to facilitate productive interactions. Experiments on $10$ surgical manipulation tasks from SurRoL, a comprehensive surgical simulation platform, demonstrate significant improvements.
  arXiv  Detail & Related papers  (2023-02-20T05:38:54Z)
• Accelerating Robot Learning of Contact-Rich Manipulations: A Curriculum Learning Study [4.045850174820418]
  This paper presents a study for accelerating robot learning of contact-rich manipulation tasks based on Curriculum Learning combined with Domain Randomization (DR) We tackle complex industrial assembly tasks with position-controlled robots, such as insertion tasks. Results also show that even when training only in simulation with toy tasks, our method can learn policies that can be transferred to the real-world robot.
  arXiv  Detail & Related papers  (2022-04-27T11:08:39Z)
• ReIL: A Framework for Reinforced Intervention-based Imitation Learning [3.0846824529023387]
  We introduce Reinforced Intervention-based Learning (ReIL), a framework consisting of a general intervention-based learning algorithm and a multi-task imitation learning model. Experimental results from real world mobile robot navigation challenges indicate that ReIL learns rapidly from sparse supervisor corrections without suffering deterioration in performance.
  arXiv  Detail & Related papers  (2022-03-29T09:30:26Z)
• Reset-Free Reinforcement Learning via Multi-Task Learning: Learning Dexterous Manipulation Behaviors without Human Intervention [67.1936055742498]
  We show that multi-task learning can effectively scale reset-free learning schemes to much more complex problems. This work shows the ability to learn dexterous manipulation behaviors in the real world with RL without any human intervention.
  arXiv  Detail & Related papers  (2021-04-22T17:38:27Z)
• Domain Adaptive Robotic Gesture Recognition with Unsupervised Kinematic-Visual Data Alignment [60.31418655784291]
  We propose a novel unsupervised domain adaptation framework which can simultaneously transfer multi-modality knowledge, i.e., both kinematic and visual data, from simulator to real robot. It remedies the domain gap with enhanced transferable features by using temporal cues in videos, and inherent correlations in multi-modal towards recognizing gesture. Results show that our approach recovers the performance with great improvement gains, up to 12.91% in ACC and 20.16% in F1score without using any annotations in real robot.
  arXiv  Detail & Related papers  (2021-03-06T09:10:03Z)
• Understanding Multi-Modal Perception Using Behavioral Cloning for Peg-In-a-Hole Insertion Tasks [21.275342989110978]
  In this paper, we investigate the merits of multiple sensor modalities when combined to learn a controller for real world assembly operation tasks. We propose a multi-step-ahead loss function to improve the performance of the behavioral cloning method.
  arXiv  Detail & Related papers  (2020-07-22T19:46:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.