SAW: Toward a Surgical Action World Model via Controllable and Scalable Video Generation

• URL: http://arxiv.org/abs/2603.13024v1
• Date: Fri, 13 Mar 2026 14:32:41 GMT
• Title: SAW: Toward a Surgical Action World Model via Controllable and Scalable Video Generation
• Abstract summary: A surgical world model capable of generating realistic surgical action videos can address fundamental challenges in surgical AI and simulation.<n>Current video generation methods require expensive annotations or complex structured intermediates as conditioning signals at inference.<n>We propose Surgical Action World (SAW) -- a step toward surgical action modeling through video diffusion conditioned on four lightweight signals.
• Score: 13.94653131033701
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A surgical world model capable of generating realistic surgical action videos with precise control over tool-tissue interactions can address fundamental challenges in surgical AI and simulation -- from data scarcity and rare event synthesis to bridging the sim-to-real gap for surgical automation. However, current video generation methods, the very core of such surgical world models, require expensive annotations or complex structured intermediates as conditioning signals at inference, limiting their scalability. Other approaches exhibit limited temporal consistency across complex laparoscopic scenes and do not possess sufficient realism. We propose Surgical Action World (SAW) -- a step toward surgical action world modeling through video diffusion conditioned on four lightweight signals: language prompts encoding tool-action context, a reference surgical scene, tissue affordance mask, and 2D tool-tip trajectories. We design a conditional video diffusion approach that reformulates video-to-video diffusion into trajectory-conditioned surgical action synthesis. The backbone diffusion model is fine-tuned on a custom-curated dataset of 12,044 laparoscopic clips with lightweight spatiotemporal conditioning signals, leveraging a depth consistency loss to enforce geometric plausibility without requiring depth at inference. SAW achieves state-of-the-art temporal consistency (CD-FVD: 199.19 vs. 546.82) and strong visual quality on held-out test data. Furthermore, we demonstrate its downstream utility for (a) surgical AI, where augmenting rare actions with SAW-generated videos improves action recognition (clipping F1-score: 20.93% to 43.14%; cutting: 0.00% to 8.33%) on real test data, and (b) surgical simulation, where rendering tool-tissue interaction videos from simulator-derived trajectory points toward a visually faithful simulation engine.

Related papers

• Surgical WAM: A World-Action Model for Data-Efficient Surgical Robot Learning [46.22753169557017]
  We introduce the Surgical World-Action Model (Surgical WAM), a unified generative model that predicts future endoscopic observations and executable surgical robot action chunks.<n>Surgical WAM first learns surgical visual dynamics from action-free video and is then fine-tuned on the fixed action-labeled budget.<n>On a suite of four simulated surgical manipulation tasks, video pretraining improves the average success rate from 63.5% to 77.8%, including an absolute gain of 20 percentage points on PegTransfer.
  arXiv  Detail & Related papers  (2026-08-11T17:59:13Z)
• SWoMo: Neuro-Symbolic World Model for Cataract Surgery Simulation [0.7880846345305875]
  We introduce SWoMo, a neuro-symbolic world model for cataract surgery simulation that decouples motion generation from visual realism.<n>We propose an inverse pairing strategy that reconstructs real surgical videos in the simulator to obtain paired simulated and real videos, which are then used to train our video diffusion model.
  arXiv  Detail & Related papers  (2026-05-15T18:27:29Z)
• Scaling Video Pretraining for Surgical Foundation Models [51.92777479821822]
  SurgRec is a scalable and reproducible pretraining recipe for surgical video understanding.<n>We curate a large corpus of 10,535 videos and 214.5M frames spanning endoscopy, laparoscopy, cataract, and robotic surgery.
  arXiv  Detail & Related papers  (2026-03-31T16:31:25Z)
• Learning Surgical Robotic Manipulation with 3D Spatial Priors [73.00031539525202]
  We introduce the Spatial Surgical Transformer (SST), an end-to-end visuomotor policy that empowers surgical robots with 3D spatial awareness.<n>Based on Surgical3D, we finetune a powerful geometric transformer to extract robust 3D latent representations from stereo endoscopes images.<n>SST achieves state-of-the-art performance and strong spatial generalization on complex surgical tasks.
  arXiv  Detail & Related papers  (2026-03-04T07:19:56Z)
• SurgVidLM: Towards Multi-grained Surgical Video Understanding with Large Language Model [67.8359850515282]
  SurgVidLM is the first video language model designed to address both full and fine-grained surgical video comprehension.<n>We show that SurgVidLM significantly outperforms state-of-the-art Vid-LLMs of comparable parameter scale in both full and fine-grained video understanding tasks.
  arXiv  Detail & Related papers  (2025-06-22T02:16:18Z)
• Mission Balance: Generating Under-represented Class Samples using Video Diffusion Models [1.5678321653327674]
  We propose a two-stage, text-based method to generate high-fidelity surgical videos for under-represented classes.<n>We evaluate our method on two downstream tasks--action recognition and intra-operative event prediction-demonstrating.
  arXiv  Detail & Related papers  (2025-05-14T23:43:29Z)
• Towards Suturing World Models: Learning Predictive Models for Robotic Surgical Tasks [0.35087986342428684]
  We introduce diffusion-based temporal models that capture the dynamics of fine-grained robotic sub-stitch actions.<n>We fine-tune two state-of-the-art video diffusion models to generate high-fidelity surgical action sequences at $ge$Lox resolution and $ge$49 frames.<n>Our experimental results demonstrate that these world models can effectively capture the dynamics of suturing, potentially enabling improved training, skill assessment tools, and autonomous surgical systems.
  arXiv  Detail & Related papers  (2025-03-16T14:51:12Z)
• Realistic Surgical Simulation from Monocular Videos [49.114370617718244]
  This paper tackles the challenge of automatically performing realistic surgical simulations from readily available surgical videos.<n>We propose SurgiSim, a novel automatic simulation system to overcome these limitations.<n> Experiments on various surgical scenarios and interactions demonstrate SurgiSim's ability to perform realistic simulation of soft tissues.
  arXiv  Detail & Related papers  (2024-12-03T10:32:41Z)
• SimuScope: Realistic Endoscopic Synthetic Dataset Generation through Surgical Simulation and Diffusion Models [1.28795255913358]
  We introduce a fully-fledged surgical simulator that automatically produces all necessary annotations for modern CAS systems.<n>It offers a more complex and realistic simulation of surgical interactions, including the dynamics between surgical instruments and deformable anatomical environments.<n>We propose a lightweight and flexible image-to-image translation method based on Stable Diffusion and Low-Rank Adaptation.
  arXiv  Detail & Related papers  (2024-12-03T09:49:43Z)
• VISAGE: Video Synthesis using Action Graphs for Surgery [34.21344214645662]
  We introduce the novel task of future video generation in laparoscopic surgery. Our proposed method, VISAGE, leverages the power of action scene graphs to capture the sequential nature of laparoscopic procedures. Results of our experiments demonstrate high-fidelity video generation for laparoscopy procedures.
  arXiv  Detail & Related papers  (2024-10-23T10:28:17Z)
• Creating a Digital Twin of Spinal Surgery: A Proof of Concept [68.37190859183663]
  Surgery digitalization is the process of creating a virtual replica of real-world surgery. We present a proof of concept (PoC) for surgery digitalization that is applied to an ex-vivo spinal surgery. We employ five RGB-D cameras for dynamic 3D reconstruction of the surgeon, a high-end camera for 3D reconstruction of the anatomy, an infrared stereo camera for surgical instrument tracking, and a laser scanner for 3D reconstruction of the operating room and data fusion.
  arXiv  Detail & Related papers  (2024-03-25T13:09:40Z)
• Scaling Up Dynamic Human-Scene Interaction Modeling [58.032368564071895]
  TRUMANS is the most comprehensive motion-captured HSI dataset currently available. It intricately captures whole-body human motions and part-level object dynamics. We devise a diffusion-based autoregressive model that efficiently generates HSI sequences of any length.
  arXiv  Detail & Related papers  (2024-03-13T15:45:04Z)
• Neural LerPlane Representations for Fast 4D Reconstruction of Deformable Tissues [52.886545681833596]
  LerPlane is a novel method for fast and accurate reconstruction of surgical scenes under a single-viewpoint setting. LerPlane treats surgical procedures as 4D volumes and factorizes them into explicit 2D planes of static and dynamic fields. LerPlane shares static fields, significantly reducing the workload of dynamic tissue modeling.
  arXiv  Detail & Related papers  (2023-05-31T14:38:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.