SWoMo: Neuro-Symbolic World Model for Cataract Surgery Simulation

• URL: http://arxiv.org/abs/2605.16530v2
• Date: Wed, 20 May 2026 08:53:25 GMT
• Title: SWoMo: Neuro-Symbolic World Model for Cataract Surgery Simulation
• Abstract summary: We introduce SWoMo, a neuro-symbolic world model for cataract surgery simulation that decouples motion generation from visual realism.<n>We propose an inverse pairing strategy that reconstructs real surgical videos in the simulator to obtain paired simulated and real videos, which are then used to train our video diffusion model.
• Score: 0.7880846345305875
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Realistic surgical simulation plays a crucial role in training novice surgeons and in the development of autonomous agents. World models can scale such simulation environments to realistic and diverse procedures by predicting future patient states conditioned on current observations and surgical actions. However, current state-of-the-art approaches often fail to satisfy key criteria required for clinical applicability, including visual realism, physically grounded interactions, and the ability to simulate scenarios beyond the training distribution. Hence, we introduce SWoMo, a neuro-symbolic world model for cataract surgery simulation that decouples motion generation from visual realism. The symbolic component, consisting of a rule-based simulator and scene graph representations, models motion dynamics and tool-tissue interactions, while a diffusion model produces realistic visual appearance, including textures and tissue deformations. We propose an inverse pairing strategy that reconstructs real surgical videos in the simulator to obtain paired simulated and real videos, which are then used to train our video diffusion model for the reverse objective of sim-to-real translation. Our experiments show both qualitative and quantitative improvements over prior work. We demonstrate that our simulator further satisfies the key criteria, including generalisation to unseen interaction geometries, improvements in downstream phase detection, and unsupervised video style transfer. The code, data, and model weights are available at: https://ssharvienkumar.github.io/SWoMo/

Related papers

• SOMA: From Surface Observations to Muscle Anatomy [55.23576061150391]
  We present SOMA, a person-specific model that infers bio-temporal muscle behavior from surface signals obtained using RGB cameras.<n>We show how our method provides anatomically grounded animations without the complexity of traditional simulations.
  arXiv  Detail & Related papers  (2026-06-08T09:20:46Z)
• SAW: Toward a Surgical Action World Model via Controllable and Scalable Video Generation [13.94653131033701]
  A surgical world model capable of generating realistic surgical action videos can address fundamental challenges in surgical AI and simulation.<n>Current video generation methods require expensive annotations or complex structured intermediates as conditioning signals at inference.<n>We propose Surgical Action World (SAW) -- a step toward surgical action modeling through video diffusion conditioned on four lightweight signals.
  arXiv  Detail & Related papers  (2026-03-13T14:32:41Z)
• DiffusionHarmonizer: Bridging Neural Reconstruction and Photorealistic Simulation with Online Diffusion Enhancer [62.18680935878919]
  We introduce DiffusionHarmonizer, an online generative enhancement framework that transforms renderings into temporally consistent outputs.<n>At its core is a single-step temporally-conditioned enhancer capable of running in online simulators on a single GPU.
  arXiv  Detail & Related papers  (2026-02-27T15:35:30Z)
• Mirage2Matter: A Physically Grounded Gaussian World Model from Video [87.9732484393686]
  We present Simulate Anything, a graphics-driven world modeling and simulation framework.<n>Our approach reconstructs real-world environments into a photorealistic scene representation using 3D Gaussian Splatting (3DGS)<n>We then leverage generative models to recover a physically realistic representation and integrate it into a simulation environment via a precision calibration target.
  arXiv  Detail & Related papers  (2026-01-24T07:43:57Z)
• Towards Suturing World Models: Learning Predictive Models for Robotic Surgical Tasks [0.35087986342428684]
  We introduce diffusion-based temporal models that capture the dynamics of fine-grained robotic sub-stitch actions.<n>We fine-tune two state-of-the-art video diffusion models to generate high-fidelity surgical action sequences at $ge$Lox resolution and $ge$49 frames.<n>Our experimental results demonstrate that these world models can effectively capture the dynamics of suturing, potentially enabling improved training, skill assessment tools, and autonomous surgical systems.
  arXiv  Detail & Related papers  (2025-03-16T14:51:12Z)
• Pre-Trained Video Generative Models as World Simulators [59.546627730477454]
  We propose Dynamic World Simulation (DWS) to transform pre-trained video generative models into controllable world simulators.<n>To achieve precise alignment between conditioned actions and generated visual changes, we introduce a lightweight, universal action-conditioned module.<n> Experiments demonstrate that DWS can be versatilely applied to both diffusion and autoregressive transformer models.
  arXiv  Detail & Related papers  (2025-02-10T14:49:09Z)
• Realistic Surgical Simulation from Monocular Videos [49.114370617718244]
  This paper tackles the challenge of automatically performing realistic surgical simulations from readily available surgical videos.<n>We propose SurgiSim, a novel automatic simulation system to overcome these limitations.<n> Experiments on various surgical scenarios and interactions demonstrate SurgiSim's ability to perform realistic simulation of soft tissues.
  arXiv  Detail & Related papers  (2024-12-03T10:32:41Z)
• SimuScope: Realistic Endoscopic Synthetic Dataset Generation through Surgical Simulation and Diffusion Models [1.28795255913358]
  We introduce a fully-fledged surgical simulator that automatically produces all necessary annotations for modern CAS systems.<n>It offers a more complex and realistic simulation of surgical interactions, including the dynamics between surgical instruments and deformable anatomical environments.<n>We propose a lightweight and flexible image-to-image translation method based on Stable Diffusion and Low-Rank Adaptation.
  arXiv  Detail & Related papers  (2024-12-03T09:49:43Z)
• SimEndoGS: Efficient Data-driven Scene Simulation using Robotic Surgery Videos via Physics-embedded 3D Gaussians [19.590481146949685]
  We introduce 3D Gaussian as a learnable representation for surgical scene, which is learned from stereo endoscopic video. We apply the Material Point Method, which is integrated with physical properties, to the 3D Gaussians to achieve realistic scene deformations. Results show that it can reconstruct and simulate surgical scenes from endoscopic videos efficiently-taking only a few minutes to reconstruct the surgical scene.
  arXiv  Detail & Related papers  (2024-05-02T02:34:19Z)
• Learning Interactive Real-World Simulators [96.5991333400566]
  We explore the possibility of learning a universal simulator of real-world interaction through generative modeling. We use the simulator to train both high-level vision-language policies and low-level reinforcement learning policies. Video captioning models can benefit from training with simulated experience, opening up even wider applications.
  arXiv  Detail & Related papers  (2023-10-09T19:42:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.