Speak, Segment, Track, Navigate: An Interactive System for Video-Guided Skull-Base Surgery

• URL: http://arxiv.org/abs/2603.16024v1
• Date: Tue, 17 Mar 2026 00:15:17 GMT
• Title: Speak, Segment, Track, Navigate: An Interactive System for Video-Guided Skull-Base Surgery
• Abstract summary: We introduce a speech-guided embodied agent framework for video-guided skull base surgery.<n>The proposed system integrates natural language interaction with real-time visual perception directly on live intraoperative video streams.<n>We evaluate the proposed system in video-guided skull base surgery scenarios and benchmark its tracking performance against a commercially available optical tracking system.
• Score: 4.97436124491469
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We introduce a speech-guided embodied agent framework for video-guided skull base surgery that dynamically executes perception and image-guidance tasks in response to surgeon queries. The proposed system integrates natural language interaction with real-time visual perception directly on live intraoperative video streams, thereby enabling surgeons to request computational assistance without disengaging from operative tasks. Unlike conventional image-guided navigation systems that rely on external optical trackers and additional hardware setup, the framework operates purely on intraoperative video. The system begins with interactive segmentation and labeling of the surgical instrument. The segmented instrument is then used as a spatial anchor that is autonomously tracked in the video stream to support downstream workflows, including anatomical segmentation, interactive registration of preoperative 3D models, monocular video-based estimation of the surgical tool pose, and support image guidance through real-time anatomical overlays.We evaluate the proposed system in video-guided skull base surgery scenarios and benchmark its tracking performance against a commercially available optical tracking system. Results demonstrate that speech-guided embodied agents can achieve competitive spatial accuracy while improving workflow integration and enabling rapid deployment of video-guided surgical systems.

Related papers

• Extend Your Horizon: A Device-Agnostic Surgical Tool Tracking Framework with Multi-View Optimization for Augmented Reality [40.023882327049684]
  Surgical navigation provides real-time guidance by estimating the pose of patient anatomy and surgical instruments.<n>Most existing approaches rely on a clear line of sight, which is difficult to maintain in dynamic operating room environments.<n>This work introduces a framework for tracking surgical instruments under occlusion by fusing multiple sensing modalities within a dynamic scene graph representation.
  arXiv  Detail & Related papers  (2026-03-09T05:37:49Z)
• Where It Moves, It Matters: Referring Surgical Instrument Segmentation via Motion [54.359489807885616]
  SurgRef is a motion-guided framework that grounds free-form language expressions in instrument motion, rather than what they look like.<n>To train and evaluate SurgRef, we present Ref-IMotion, a diverse, multi-institutional video dataset with densetemporal masks and rich motion expressions.
  arXiv  Detail & Related papers  (2026-01-18T02:14:08Z)
• Sound Source Localization for Spatial Mapping of Surgical Actions in Dynamic Scenes [0.5146940511526402]
  This work aims to enhance surgical scene representations by integrating 3D acoustic information.<n>We propose a novel framework for generating 4D audio-visual representations of surgical scenes.<n>The proposed framework enables richer contextual understanding and provides a foundation for future intelligent surgical systems.
  arXiv  Detail & Related papers  (2025-10-28T11:55:45Z)
• SCOPE: Speech-guided COllaborative PErception Framework for Surgical Scene Segmentation [4.97436124491469]
  We introduce a speech-guided collaborative perception framework that integrates reasoning capabilities of large language model (LLM) with perception capabilities of open-set VFMs.<n>A key component of this framework is a collaborative perception agent, which generates top candidates of VFM-generated segmentation.<n> instruments themselves serve as interactive pointers to label additional elements of the surgical scene.
  arXiv  Detail & Related papers  (2025-09-12T23:36:52Z)
• EndoChat: Grounded Multimodal Large Language Model for Endoscopic Surgery [52.992415247012296]
  We introduce EndoChat to address various dialogue paradigms and subtasks in surgical scene understanding.<n>Our model achieves state-of-the-art performance across five dialogue paradigms and eight surgical scene understanding tasks.
  arXiv  Detail & Related papers  (2025-01-20T09:12:06Z)
• Online 3D reconstruction and dense tracking in endoscopic videos [5.667206318889122]
  3D scene reconstruction from stereo endoscopic video data is crucial for advancing surgical interventions. We present an online framework for online, dense 3D scene reconstruction and tracking, aimed at enhancing surgical scene understanding and assisting interventions.
  arXiv  Detail & Related papers  (2024-09-09T19:58:42Z)
• Learning Multi-modal Representations by Watching Hundreds of Surgical Video Lectures [50.09187683845788]
  Recent advancements in surgical computer vision applications have been driven by vision-only models.<n>These methods rely on manually annotated surgical videos to predict a fixed set of object categories.<n>In this work, we put forward the idea that the surgical video lectures available through open surgical e-learning platforms can provide effective vision and language supervisory signals.
  arXiv  Detail & Related papers  (2023-07-27T22:38:12Z)
• Live image-based neurosurgical guidance and roadmap generation using unsupervised embedding [53.992124594124896]
  We present a method for live image-only guidance leveraging a large data set of annotated neurosurgical videos. A generated roadmap encodes the common anatomical paths taken in surgeries in the training set. We trained and evaluated the proposed method with a data set of 166 transsphenoidal adenomectomy procedures.
  arXiv  Detail & Related papers  (2023-03-31T12:52:24Z)
• Multimodal Semantic Scene Graphs for Holistic Modeling of Surgical Procedures [70.69948035469467]
  We take advantage of the latest computer vision methodologies for generating 3D graphs from camera views. We then introduce the Multimodal Semantic Graph Scene (MSSG) which aims at providing unified symbolic and semantic representation of surgical procedures.
  arXiv  Detail & Related papers  (2021-06-09T14:35:44Z)
• Using Computer Vision to Automate Hand Detection and Tracking of Surgeon Movements in Videos of Open Surgery [8.095095522269352]
  We leverage advances in computer vision to introduce an automated approach to video analysis of surgical execution. A state-of-the-art convolutional neural network architecture for object detection was used to detect operating hands in open surgery videos. Our model's spatial detections of operating hands significantly outperforms the detections achieved using pre-existing hand-detection datasets.
  arXiv  Detail & Related papers  (2020-12-13T03:10:09Z)
• Relational Graph Learning on Visual and Kinematics Embeddings for Accurate Gesture Recognition in Robotic Surgery [84.73764603474413]
  We propose a novel online approach of multi-modal graph network (i.e., MRG-Net) to dynamically integrate visual and kinematics information. The effectiveness of our method is demonstrated with state-of-the-art results on the public JIGSAWS dataset.
  arXiv  Detail & Related papers  (2020-11-03T11:00:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.