Surgical Vision World Model

• URL: http://arxiv.org/abs/2503.02904v1
• Date: Mon, 03 Mar 2025 10:55:52 GMT
• Title: Surgical Vision World Model
• Authors: Saurabh Koju, Saurav Bastola, Prashant Shrestha, Sanskar Amgain, Yash Raj Shrestha, Rudra P. K. Poudel, Binod Bhattarai,
• Abstract summary: We propose the first surgical vision world model to generate action-controllable surgical data.<n>The proposed model can generate action-controllable surgical data and the architecture design is verified.
• Score: 7.227638707410672
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Realistic and interactive surgical simulation has the potential to facilitate crucial applications, such as medical professional training and autonomous surgical agent training. In the natural visual domain, world models have enabled action-controlled data generation, demonstrating the potential to train autonomous agents in interactive simulated environments when large-scale real data acquisition is infeasible. However, such works in the surgical domain have been limited to simplified computer simulations, and lack realism. Furthermore, existing literature in world models has predominantly dealt with action-labeled data, limiting their applicability to real-world surgical data, where obtaining action annotation is prohibitively expensive. Inspired by the recent success of Genie in leveraging unlabeled video game data to infer latent actions and enable action-controlled data generation, we propose the first surgical vision world model. The proposed model can generate action-controllable surgical data and the architecture design is verified with extensive experiments on the unlabeled SurgToolLoc-2022 dataset. Codes and implementation details are available at https://github.com/bhattarailab/Surgical-Vision-World-Model

Related papers

• AdaWorld: Learning Adaptable World Models with Latent Actions [76.50869178593733]
  We propose AdaWorld, an innovative world model learning approach that enables efficient adaptation. Key idea is to incorporate action information during the pretraining of world models. We then develop an autoregressive world model that conditions on these latent actions.
  arXiv  Detail & Related papers  (2025-03-24T17:58:15Z)
• Towards Suturing World Models: Learning Predictive Models for Robotic Surgical Tasks [0.35087986342428684]
  We introduce diffusion-based temporal models that capture the dynamics of fine-grained robotic sub-stitch actions. We fine-tune two state-of-the-art video diffusion models to generate high-fidelity surgical action sequences at $ge$Lox resolution and $ge$49 frames. Our experimental results demonstrate that these world models can effectively capture the dynamics of suturing, potentially enabling improved training, skill assessment tools, and autonomous surgical systems.
  arXiv  Detail & Related papers  (2025-03-16T14:51:12Z)
• Pre-Trained Video Generative Models as World Simulators [59.546627730477454]
  We propose Dynamic World Simulation (DWS) to transform pre-trained video generative models into controllable world simulators.<n>To achieve precise alignment between conditioned actions and generated visual changes, we introduce a lightweight, universal action-conditioned module.<n> Experiments demonstrate that DWS can be versatilely applied to both diffusion and autoregressive transformer models.
  arXiv  Detail & Related papers  (2025-02-10T14:49:09Z)
• SimuScope: Realistic Endoscopic Synthetic Dataset Generation through Surgical Simulation and Diffusion Models [1.28795255913358]
  We introduce a fully-fledged surgical simulator that automatically produces all necessary annotations for modern CAS systems.<n>It offers a more complex and realistic simulation of surgical interactions, including the dynamics between surgical instruments and deformable anatomical environments.<n>We propose a lightweight and flexible image-to-image translation method based on Stable Diffusion and Low-Rank Adaptation.
  arXiv  Detail & Related papers  (2024-12-03T09:49:43Z)
• Polaris: Open-ended Interactive Robotic Manipulation via Syn2Real Visual Grounding and Large Language Models [53.22792173053473]
  We introduce an interactive robotic manipulation framework called Polaris. Polaris integrates perception and interaction by utilizing GPT-4 alongside grounded vision models. We propose a novel Synthetic-to-Real (Syn2Real) pose estimation pipeline.
  arXiv  Detail & Related papers  (2024-08-15T06:40:38Z)
• Realistic Surgical Image Dataset Generation Based On 3D Gaussian Splatting [3.5351922399745166]
  This research introduces a novel method that employs 3D Gaussian Splatting to generate synthetic surgical datasets. We developed a data recording system capable of acquiring images alongside tool and camera poses in a surgical scene. Using this pose data, we synthetically replicate the scene, thereby enabling direct comparisons of the synthetic image quality.
  arXiv  Detail & Related papers  (2024-07-20T11:20:07Z)
• Realistic Data Generation for 6D Pose Estimation of Surgical Instruments [4.226502078427161]
  6D pose estimation of surgical instruments is critical to enable the automatic execution of surgical maneuvers. In household and industrial settings, synthetic data, generated with 3D computer graphics software, has been shown as an alternative to minimize annotation costs. We propose an improved simulation environment for surgical robotics that enables the automatic generation of large and diverse datasets.
  arXiv  Detail & Related papers  (2024-06-11T14:59:29Z)
• Learning Interactive Real-World Simulators [96.5991333400566]
  We explore the possibility of learning a universal simulator of real-world interaction through generative modeling. We use the simulator to train both high-level vision-language policies and low-level reinforcement learning policies. Video captioning models can benefit from training with simulated experience, opening up even wider applications.
  arXiv  Detail & Related papers  (2023-10-09T19:42:22Z)
• GAMMA: Generalizable Articulation Modeling and Manipulation for Articulated Objects [53.965581080954905]
  We propose a novel framework of Generalizable Articulation Modeling and Manipulating for Articulated Objects (GAMMA) GAMMA learns both articulation modeling and grasp pose affordance from diverse articulated objects with different categories. Results show that GAMMA significantly outperforms SOTA articulation modeling and manipulation algorithms in unseen and cross-category articulated objects.
  arXiv  Detail & Related papers  (2023-09-28T08:57:14Z)
• RT-1: Robotics Transformer for Real-World Control at Scale [98.09428483862165]
  We present a model class, dubbed Robotics Transformer, that exhibits promising scalable model properties. We verify our conclusions in a study of different model classes and their ability to generalize as a function of the data size, model size, and data diversity based on a large-scale data collection on real robots performing real-world tasks.
  arXiv  Detail & Related papers  (2022-12-13T18:55:15Z)
• Surgical Visual Domain Adaptation: Results from the MICCAI 2020 SurgVisDom Challenge [9.986124942784969]
  This work seeks to explore the potential for visual domain adaptation in surgery to overcome data privacy concerns. In particular, we propose to use video from virtual reality (VR) simulations of surgical exercises to develop algorithms to recognize tasks in a clinical-like setting. We present the performance of the different approaches to solve visual domain adaptation developed by challenge participants.
  arXiv  Detail & Related papers  (2021-02-26T18:45:28Z)
• Recurrent and Spiking Modeling of Sparse Surgical Kinematics [0.8458020117487898]
  A growing number of studies have used machine learning to analyze video and kinematic data captured from surgical robots. In this study, we explore the possibility of using only kinematic data to predict surgeons of similar skill levels. We report that it is possible to identify surgical fellows receiving near perfect scores in the simulation exercises based on their motion characteristics alone.
  arXiv  Detail & Related papers  (2020-05-12T15:41:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.