Surgical WAM: A World-Action Model for Data-Efficient Surgical Robot Learning

• URL: http://arxiv.org/abs/2608.11204v1
• Date: Tue, 11 Aug 2026 17:59:13 GMT
• Title: Surgical WAM: A World-Action Model for Data-Efficient Surgical Robot Learning
• Abstract summary: We introduce the Surgical World-Action Model (Surgical WAM), a unified generative model that predicts future endoscopic observations and executable surgical robot action chunks.<n>Surgical WAM first learns surgical visual dynamics from action-free video and is then fine-tuned on the fixed action-labeled budget.<n>On a suite of four simulated surgical manipulation tasks, video pretraining improves the average success rate from 63.5% to 77.8%, including an absolute gain of 20 percentage points on PegTransfer.
• Score: 46.22753169557017
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Learning reliable surgical manipulation policies is bottlenecked by the scarcity of action-labeled demonstrations: teleoperated surgical robot (e.g., dVRK) trajectories with synchronized kinematics are costly to collect, while surgical tasks demand precise contact handling, long-horizon reasoning, and bimanual coordination. Endoscopic video is comparatively inexpensive and abundant relative to synchronized video--kinematics trajectories, and a natural way to exploit it is to learn world models of surgical scenes. However, existing surgical world models use video primarily for simulation or policy evaluation, and rarely translate the learned dynamics into closed-loop control. This gap raises our central question: under a fixed budget of action-labeled demonstrations, does action-free video pretraining improve closed-loop surgical manipulation? To answer it, we introduce the Surgical World-Action Model (Surgical WAM), a unified generative model built on Cosmos Policy that jointly predicts future endoscopic observations and executable surgical robot action chunks. Surgical WAM first learns surgical visual dynamics from action-free video and is then fine-tuned on the fixed action-labeled budget; at deployment, it acts as a closed-loop, receding-horizon controller that executes a short prefix of each predicted action chunk and replans from the resulting observation. On a suite of four simulated surgical manipulation tasks, video pretraining improves the average success rate from 63.5% to 77.8%, including an absolute gain of 20 percentage points on PegTransfer, with the largest improvements on contact-rich and bimanual tasks. These results demonstrate that action-free video provides transferable visual dynamics priors for learning surgical robot control with limited action supervision, positioning data-efficient video pretraining as a practical path toward scaling up surgical robot learning.

Related papers

• SurgVIL: Scaling Surgical Robot Imitation Learning with Open-source Surgical Videos [3.1495623798952823]
  SurgVIL is a framework for scaling surgical robot imitation learning using open-source surgical videos.<n>We evaluate SurgVIL on two da Vinci robot tasks: needle pick-up and cholecystectomy cutting.
  arXiv  Detail & Related papers  (2026-08-17T03:33:03Z)
• SurgWorld: Learning Surgical Robot Policies from Videos via World Modeling [19.99022199561975]
  SurgWorld is a world model designed for surgical physical AI.<n>SurgeWorld generates diverse, generalizable and realistic surgery videos.<n>We are the first to use an inverse dynamics model to infer pseudokinematics from synthetic surgical videos.
  arXiv  Detail & Related papers  (2025-12-29T03:03:00Z)
• SurgiPose: Estimating Surgical Tool Kinematics from Monocular Video for Surgical Robot Learning [5.304104136888954]
  A promising source of large-scale surgical demonstrations is monocular surgical videos available online.<n>We propose SurgiPose, a differentiable rendering based approach to estimate kinematic information from monocular surgical videos.<n>Our results show that policies trained on estimated kinematics achieve comparable success rates to those trained on ground truth data.
  arXiv  Detail & Related papers  (2025-12-19T21:15:26Z)
• FreeAction: Training-Free Techniques for Enhanced Fidelity of Trajectory-to-Video Generation [50.39748673817223]
  We introduce two training-free, inference-time techniques that fully exploit explicit action parameters in robot video generation.<n>First, action-scaled classifier-free guidance dynamically modulates guidance strength in proportion to action magnitude, enhancing controllability over motion intensity.<n>Second, action-scaled noise truncation adjusts the distribution of initially sampled noise to better align with the desired motion dynamics.
  arXiv  Detail & Related papers  (2025-09-29T03:30:40Z)
• Physical Autoregressive Model for Robotic Manipulation without Action Pretraining [65.8971623698511]
  We build upon autoregressive video generation models to propose a Physical Autoregressive Model (PAR)<n>PAR leverages the world knowledge embedded in video pretraining to understand physical dynamics without requiring action pretraining.<n>Experiments on the ManiSkill benchmark show that PAR achieves a 100% success rate on the PushCube task.
  arXiv  Detail & Related papers  (2025-08-13T13:54:51Z)
• SurgVidLM: Towards Multi-grained Surgical Video Understanding with Large Language Model [67.8359850515282]
  SurgVidLM is the first video language model designed to address both full and fine-grained surgical video comprehension.<n>We show that SurgVidLM significantly outperforms state-of-the-art Vid-LLMs of comparable parameter scale in both full and fine-grained video understanding tasks.
  arXiv  Detail & Related papers  (2025-06-22T02:16:18Z)
• Towards Suturing World Models: Learning Predictive Models for Robotic Surgical Tasks [0.35087986342428684]
  We introduce diffusion-based temporal models that capture the dynamics of fine-grained robotic sub-stitch actions.<n>We fine-tune two state-of-the-art video diffusion models to generate high-fidelity surgical action sequences at $ge$Lox resolution and $ge$49 frames.<n>Our experimental results demonstrate that these world models can effectively capture the dynamics of suturing, potentially enabling improved training, skill assessment tools, and autonomous surgical systems.
  arXiv  Detail & Related papers  (2025-03-16T14:51:12Z)
• Moto: Latent Motion Token as the Bridging Language for Learning Robot Manipulation from Videos [101.26467307473638]
  We introduce Moto, which converts video content into latent Motion Token sequences by a Latent Motion Tokenizer.<n>We pre-train Moto-GPT through motion token autoregression, enabling it to capture diverse visual motion knowledge.<n>To transfer learned motion priors to real robot actions, we implement a co-fine-tuning strategy that seamlessly bridges latent motion token prediction and real robot control.
  arXiv  Detail & Related papers  (2024-12-05T18:57:04Z)
• VISAGE: Video Synthesis using Action Graphs for Surgery [34.21344214645662]
  We introduce the novel task of future video generation in laparoscopic surgery. Our proposed method, VISAGE, leverages the power of action scene graphs to capture the sequential nature of laparoscopic procedures. Results of our experiments demonstrate high-fidelity video generation for laparoscopy procedures.
  arXiv  Detail & Related papers  (2024-10-23T10:28:17Z)
• Future Frame Prediction for Robot-assisted Surgery [57.18185972461453]
  We propose a ternary prior guided variational autoencoder (TPG-VAE) model for future frame prediction in robotic surgical video sequences. Besides content distribution, our model learns motion distribution, which is novel to handle the small movements of surgical tools.
  arXiv  Detail & Related papers  (2021-03-18T15:12:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.