SurgWMBench: A Vision-Based Benchmark for World-Modeling Surgical Instrument Motion Planning

• URL: http://arxiv.org/abs/2608.08070v1
• Date: Sat, 08 Aug 2026 11:30:50 GMT
• Title: SurgWMBench: A Vision-Based Benchmark for World-Modeling Surgical Instrument Motion Planning
• Abstract summary: We introduce SurgWMBench, a vision-based benchmark for short-horizon surgical motion planning and dynamics prediction.<n>World models offer a natural framework for jointly modeling visual state transitions and instrument motion dynamics.
• Score: 20.03554949764767
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Reliable surgical planning requires models that move beyond recognizing the current surgical step or imitating expert demonstrations, and instead anticipate how instrument motion reshapes subsequent operative states. Most surgical video understanding methods focus on recognizing phases, actions, or workflow states, while providing limited support for explicitly modeling instrument motion. Conversely, existing tool motion prediction methods can forecast instrument trajectories, but they generally do not capture the coupled evolution of future surgical video states. World models offer a natural framework for jointly modeling visual state transitions and instrument motion dynamics. However, existing surgical world model studies remain largely centered on visual generation quality, relying on generation-oriented metrics such as FVD and CD-FVD. These metrics are poorly aligned with instrument motion planning, as they do not directly measure whether predicted trajectories are geometrically accurate, temporally coherent, or actionable for downstream planning. This limitation is partly structural, since the field lacks public datasets and standardized evaluation protocols that provide the benchmarking infrastructure needed to assess motion-centric capabilities in surgical world models. In this paper, we introduce SurgWMBench, a vision-based benchmark for short-horizon surgical motion planning and dynamics prediction. Given intraoperative image sequences and historical instrument trajectory, SurgWMBench evaluates both near-future instrument motion prediction and stability under continuous rollout or input perturbations.

Related papers

• CST-WM: A Causally Structured World Model for Embodied Visual Tracking [9.902564083063973]
  Embodied visual tracking requires a robot to choose actions that preserve or recover future evidence of a moving target.<n>CST-WM decomposes the latent state into target-evidence, robot, and observation branches.<n>It supports both stable following and temporary target re-acquisition in one planning framework.
  arXiv  Detail & Related papers  (2026-09-05T23:24:32Z)
• Towards Surgical World-Action Modeling: A Preliminary Joint Visual-Trajectory Forecasting for Surgical Motion Planning [37.17021354294023]
  We present a joint visual-trajectory world-action model that simultaneously forecasts future visual states and instrument trajectories from historical surgical observations.<n>We observe progressive visual degradation and accumulated trajectory errors over longer prediction horizons.
  arXiv  Detail & Related papers  (2026-08-20T17:18:02Z)
• EndoWAM: A Grounded World-Action Model for Generalizable Endoscopic Navigation [12.580242508201827]
  We present EndoWAM, to our knowledge, the first WAM for generalizable robotic endoscopic navigation.<n>Future grounding predicts task-relevant target regions in future observations from intermediate denoising features of a video world model.<n>This design injects target-aware supervision into predictive dynamics modeling, improving robustness to visual degradation and viewpoint changes.
  arXiv  Detail & Related papers  (2026-08-02T13:11:22Z)
• From World Action Models to Embodied Brains: A Roadmap for Open-World Physical Intelligence [54.62791372989525]
  Action models, vision-language--action policies, and world models have advanced this goal.<n>World Action Models (WAMs) are particularly promising because they connect candidate interventions with predicted consequences.<n>We review the evolution toward WAMs and organize these limitations into three coupled gaps: model roles and representations, objectives and standardization, and system composition.
  arXiv  Detail & Related papers  (2026-07-13T15:22:56Z)
• From Imagined Futures to Executable Actions: Mixture of Latent Actions for Robot Manipulation [88.39072412680633]
  We propose MoLA, a control-oriented interface that transforms imagined future videos into executable representations.<n>We evaluate our approach on simulated benchmarks and real-world robot manipulation tasks.
  arXiv  Detail & Related papers  (2026-05-12T14:15:16Z)
• FutureVLA: Joint Visuomotor Prediction for Vision-Language-Action Model [73.03346643967309]
  We argue that effective joint visuomotor predictive modeling requires both temporal continuity and visually-conditioned supervision decoupling.<n>FutureVLA is designed to extract joint visuomotor embeddings by first decoupling visual and motor information.<n>In the post-training stage, we employ a latent embeddings alignment strategy, enabling diverse downstream VLA models to internalize these temporal priors.
  arXiv  Detail & Related papers  (2026-03-11T12:39:55Z)
• Vision-Centric 4D Occupancy Forecasting and Planning via Implicit Residual World Models [28.777224599594717]
  Implicit Residual World Model focuses on modeling the current state and evolution of the world.<n> IR-WM achieves top performance in both 4D occupancy forecasting and trajectory planning.
  arXiv  Detail & Related papers  (2025-10-19T06:45:37Z)
• Ego-centric Predictive Model Conditioned on Hand Trajectories [52.531681772560724]
  In egocentric scenarios, anticipating both the next action and its visual outcome is essential for understanding human-object interactions.<n>We propose a unified two-stage predictive framework that jointly models action and visual future in egocentric scenarios.<n>Our approach is the first unified model designed to handle both egocentric human activity understanding and robotic manipulation tasks.
  arXiv  Detail & Related papers  (2025-08-27T13:09:55Z)
• Human locomotor control timescales depend on the environmental context and sensory input modality [37.48294298569551]
  We present a unified data-driven framework to quantify the control timescales.<n>We apply this framework across tasks including walking and running.<n>Our framework reveals the factors that influence locomotor foot placement control timescales.
  arXiv  Detail & Related papers  (2025-03-20T16:57:15Z)
• Multimodal Semantic Scene Graphs for Holistic Modeling of Surgical Procedures [70.69948035469467]
  We take advantage of the latest computer vision methodologies for generating 3D graphs from camera views. We then introduce the Multimodal Semantic Graph Scene (MSSG) which aims at providing unified symbolic and semantic representation of surgical procedures.
  arXiv  Detail & Related papers  (2021-06-09T14:35:44Z)
• Future Frame Prediction for Robot-assisted Surgery [57.18185972461453]
  We propose a ternary prior guided variational autoencoder (TPG-VAE) model for future frame prediction in robotic surgical video sequences. Besides content distribution, our model learns motion distribution, which is novel to handle the small movements of surgical tools.
  arXiv  Detail & Related papers  (2021-03-18T15:12:06Z)
• daVinciNet: Joint Prediction of Motion and Surgical State in Robot-Assisted Surgery [13.928484202934651]
  We propose daVinciNet - an end-to-end dual-task model for robot motion and surgical state predictions. Our model achieves up to 93.85% short-term (0.5s) and 82.11% long-term (2s) state prediction accuracy, as well as 1.07mm short-term and 5.62mm long-term trajectory prediction error.
  arXiv  Detail & Related papers  (2020-09-24T20:28:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.