SurgVIL: Scaling Surgical Robot Imitation Learning with Open-source Surgical Videos

• URL: http://arxiv.org/abs/2608.16058v1
• Date: Mon, 17 Aug 2026 03:33:03 GMT
• Title: SurgVIL: Scaling Surgical Robot Imitation Learning with Open-source Surgical Videos
• Abstract summary: SurgVIL is a framework for scaling surgical robot imitation learning using open-source surgical videos.<n>We evaluate SurgVIL on two da Vinci robot tasks: needle pick-up and cholecystectomy cutting.
• Score: 3.1495623798952823
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Learning-based surgical robot autonomy requires large-scale demonstrations with synchronized videos and robot actions, but such data are exceedingly rare in clinical or realistic tissue settings because robot kinematics are typically inaccessible outside controlled research systems. In contrast, phantom data collected on research platforms provide accurate action labels but lack the visual diversity of real tissue. We propose SurgVIL, a framework for scaling surgical robot imitation learning using open-source surgical videos. SurgVIL combines kinematically labeled phantom robot demonstrations with surgical videos from open-source datasets and online sources for policy learning. Since these videos lack robot motion labels, we estimate approximate kinematics as weak supervision. We evaluate SurgVIL on two da Vinci robot tasks: needle pick-up and cholecystectomy cutting. Across ACT, $π_0$, and GR00T-H backbones, adding surgical videos substantially improves generalization to real-tissue and out-of-distribution settings, suggesting a scalable path from phantom training toward generalizable surgical robot policies.

Related papers

• Surgical WAM: A World-Action Model for Data-Efficient Surgical Robot Learning [46.22753169557017]
  We introduce the Surgical World-Action Model (Surgical WAM), a unified generative model that predicts future endoscopic observations and executable surgical robot action chunks.<n>Surgical WAM first learns surgical visual dynamics from action-free video and is then fine-tuned on the fixed action-labeled budget.<n>On a suite of four simulated surgical manipulation tasks, video pretraining improves the average success rate from 63.5% to 77.8%, including an absolute gain of 20 percentage points on PegTransfer.
  arXiv  Detail & Related papers  (2026-08-11T17:59:13Z)
• Ego2Robot: Scalable Robot Data Synthesis from Egocentric Human Data [60.94418315274988]
  Ego2Robot supports both curated and in-the-wild videos, producing 18,561 hours of robot training data spanning 15 robot morphologies.<n>Joint pretraining on Ego2Robot-synthesized and robot data consistently improves out-of-distribution generalization across multiple types.
  arXiv  Detail & Related papers  (2026-08-03T17:52:26Z)
• SurgWorld: Learning Surgical Robot Policies from Videos via World Modeling [19.99022199561975]
  SurgWorld is a world model designed for surgical physical AI.<n>SurgeWorld generates diverse, generalizable and realistic surgery videos.<n>We are the first to use an inverse dynamics model to infer pseudokinematics from synthetic surgical videos.
  arXiv  Detail & Related papers  (2025-12-29T03:03:00Z)
• SurgiPose: Estimating Surgical Tool Kinematics from Monocular Video for Surgical Robot Learning [5.304104136888954]
  A promising source of large-scale surgical demonstrations is monocular surgical videos available online.<n>We propose SurgiPose, a differentiable rendering based approach to estimate kinematic information from monocular surgical videos.<n>Our results show that policies trained on estimated kinematics achieve comparable success rates to those trained on ground truth data.
  arXiv  Detail & Related papers  (2025-12-19T21:15:26Z)
• Large Video Planner Enables Generalizable Robot Control [117.49024534548319]
  General-purpose robots require decision-making models that generalize across diverse tasks and environments.<n>Recent works build robot foundation models by extending multimodal large language models (LMs) with action outputs, creating vision--action (VLA) systems.<n>We explore an alternative paradigm of using large-scale video pretraining as a primary modality for building robot foundation models.
  arXiv  Detail & Related papers  (2025-12-17T18:35:54Z)
• H2R-Grounder: A Paired-Data-Free Paradigm for Translating Human Interaction Videos into Physically Grounded Robot Videos [58.006918399913665]
  We propose a video-to-video translation framework that converts ordinary human-object interaction videos into motion-consistent robot manipulation videos.<n>Our approach does not require any paired human-robot videos for training only a set of unpaired robot videos, making the system easy to scale.<n>At test time, we apply the same process to human videos (inpainting the person and overlaying human pose cues) and generate high-quality robot videos that mimic the human's actions.
  arXiv  Detail & Related papers  (2025-12-10T07:59:45Z)
• VidBot: Learning Generalizable 3D Actions from In-the-Wild 2D Human Videos for Zero-Shot Robotic Manipulation [53.63540587160549]
  VidBot is a framework enabling zero-shot robotic manipulation using learned 3D affordance from in-the-wild monocular RGB-only human videos.<n> VidBot paves the way for leveraging everyday human videos to make robot learning more scalable.
  arXiv  Detail & Related papers  (2025-03-10T10:04:58Z)
• Future Frame Prediction for Robot-assisted Surgery [57.18185972461453]
  We propose a ternary prior guided variational autoencoder (TPG-VAE) model for future frame prediction in robotic surgical video sequences. Besides content distribution, our model learns motion distribution, which is novel to handle the small movements of surgical tools.
  arXiv  Detail & Related papers  (2021-03-18T15:12:06Z)
• Recurrent and Spiking Modeling of Sparse Surgical Kinematics [0.8458020117487898]
  A growing number of studies have used machine learning to analyze video and kinematic data captured from surgical robots. In this study, we explore the possibility of using only kinematic data to predict surgeons of similar skill levels. We report that it is possible to identify surgical fellows receiving near perfect scores in the simulation exercises based on their motion characteristics alone.
  arXiv  Detail & Related papers  (2020-05-12T15:41:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.