Towards Unified Surgical Scene Understanding:Bridging Reasoning and Grounding via MLLMs

• URL: http://arxiv.org/abs/2605.13530v1
• Date: Wed, 13 May 2026 13:42:23 GMT
• Title: Towards Unified Surgical Scene Understanding:Bridging Reasoning and Grounding via MLLMs
• Abstract summary: SurgMLLM is a unified surgical scene understanding framework.<n>It bridges high-level reasoning and low-level visual grounding within a single model.<n>Extensive experiments show that SurgMLLM significantly advances surgical scene understanding.
• Score: 32.18549469001096
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Surgical scene understanding is a cornerstone of computer-assisted intervention. While recent advances, particularly in surgical image segmentation, have driven progress, real-world clinical applications require a more holistic understanding that jointly captures procedural context, semantic reasoning, and precise visual grounding. However, existing approaches typically address these components in isolation, leading to fragmented representations and limited semantic consistency. To address this limitation, we propose SurgMLLM, a unified surgical scene understanding framework that bridges high-level reasoning and low-level visual grounding within a single model. Given surgical videos, SurgMLLM fine-tunes a multimodal large language model (MLLM) to support structured interpretability reasoning, which is used to jointly model phases, instrument-verb-target (IVT) triplets, and triplet-entity segmentation tokens. These tokens are then temporally aggregated and serve as prompts for a segmentation network, enabling accurate pixel-wise grounding of triplet instruments and targets. The entire framework is trained end-to-end with a unified objective that couples language-based reasoning supervision with visual grounding losses, promoting coherent cross-task learning and clinically consistent scene representations. To facilitate unified evaluation, we introduce CholecT45-Scene, extending CholecT45 dataset with 64,299 frames of pixel-level mask annotations for instruments and targets, aligned with existing triplet labels. Extensive experiments show that SurgMLLM significantly advances surgical scene understanding, improving the primary triplet recognition metric AP_IVT from 40.7% to 46.0% and consistently outperforming prior methods in phase recognition and segmentation. These results highlight the effectiveness of unified reasoning-and-grounding for reliable, context-aware surgical assistance.

Related papers

• SLIP: Segmentation with Low-latency Interactive Prompting for 3D Medical Images [58.57521567704563]
  We present SLIP, an end-to-end trainable framework for interactive 3D medical image segmentation.<n>By separating image representation from interactive reasoning, SLIP substantially reduces interaction latency.<n>SLIP achieves SOTA interactive segmentation performance across 13 public datasets.
  arXiv  Detail & Related papers  (2026-07-24T14:12:02Z)
• HTT-Net: Hierarchical Text-guided Transition Modeling for Surgical Video Phase Recognition [22.19338002382572]
  We propose HTT-Net, a Hierarchical Text-guided Transition modeling Network for surgical video phase recognition.<n>Key idea is to introduce structured surgical semantic knowledge into phase-aware segment construction and semantic refinement.<n>Experiments on Cholec80 and LCRS-100 demonstrate the effectiveness of HTT-Net for robust surgical video phase recognition.
  arXiv  Detail & Related papers  (2026-07-18T11:49:40Z)
• HyperVLP: Enhancing Hierarchical Surgical Video-Language Pre-training in Hyperbolic Space [50.603248889968306]
  We propose a hyperbolic surgical video-language pre-training framework that explicitly preserves the hierarchical structure.<n>Experiments on multiple surgical benchmarks show consistent gains in zero- and few-shot phase recognition.
  arXiv  Detail & Related papers  (2026-06-30T07:21:06Z)
• Object Tokens as a Bridge Between Segmentation and Visual Question Answering in Robotic Surgery [1.4146817306459933]
  We propose a unified framework that jointly performs pixel-level segmentation and visual question answering within a single framework.<n>We evaluate the proposed method on the private RAMIE (Robot-Assisted Minimally Invasive Esophagectomy) dataset and the public EndoVis18 dataset.<n>Results demonstrate that incorporating context-aware object tokens into vision-language models improves fine-grained surgical scene understanding.
  arXiv  Detail & Related papers  (2026-06-14T15:30:16Z)
• From Drop-off to Recovery: A Mechanistic Analysis of Segmentation in MLLMs [50.185593677108436]
  Multimodal Large Language Models (MLLMs) are increasingly applied to pixel-level vision tasks, yet their intrinsic capacity for spatial understanding remains poorly understood.<n>We investigate segmentation capacity through a layerwise linear probing evaluation across the entire MLLM pipeline: vision encoder, adapter, and LLM.
  arXiv  Detail & Related papers  (2026-03-18T00:22:15Z)
• GroundedSurg: A Multi-Procedure Benchmark for Language-Conditioned Surgical Tool Segmentation [1.9981885081131854]
  We introduce GroundedSurg, the first language-conditioned, instance-level surgical grounding benchmark.<n>The dataset spans ophthalmic, laparoscopic, robotic, and open procedures, encompassing diverse instrument types, imaging conditions, and operative complexities.
  arXiv  Detail & Related papers  (2026-03-01T13:49:53Z)
• TGC-Net: A Structure-Aware and Semantically-Aligned Framework for Text-Guided Medical Image Segmentation [56.09179939570486]
  We propose TGC-Net, a CLIP-based framework focusing on parameter-efficient, task-specific adaptations.<n>TGC-Net achieves state-of-the-art performance with substantially fewer trainable parameters, including notable Dice gains on challenging benchmarks.
  arXiv  Detail & Related papers  (2025-12-24T12:06:26Z)
• Grounding Surgical Action Triplets with Instrument Instance Segmentation: A Dataset and Target-Aware Fusion Approach [16.569535111037315]
  We present CholecTriplet-Seg, a large-scale dataset containing over 30,000 annotated frames, linking instrument instance masks with action verb and anatomical target annotations, and establishing the first benchmark for strongly supervised, instance-level triplet grounding and evaluation.<n>We also propose TargetFusionNet, a novel architecture that extends Mask2Former with a target-aware fusion mechanism to address the challenge of accurate anatomical target prediction by fusing weak anatomy priors with instrument instance queries.
  arXiv  Detail & Related papers  (2025-11-01T17:45:40Z)
• Surgical Video Understanding with Label Interpolation [3.880707330499936]
  Robot-assisted surgery (RAS) has become a critical paradigm in modern surgery, promoting patient recovery and reducing the burden on surgeons.<n>Previous studies have predominantly focused on single-task approaches, but real surgical scenes involve complex temporal dynamics and diverse instrument interactions.<n>We propose a novel framework that combines optical flow-based segmentation label with multi-task learning.
  arXiv  Detail & Related papers  (2025-09-23T08:49:07Z)
• MedReasoner: Reinforcement Learning Drives Reasoning Grounding from Clinical Thought to Pixel-Level Precision [16.34920560168635]
  We first define Unified Medical Reasoning Grounding (UMRG), a novel vision-language task.<n>Second, we release U-MRG-14K, a dataset of 14K samples featuring pixel-level masks alongside implicit clinical queries and reasoning traces.<n>Third, we introduce MedReasoner, a modular framework that distinctly separates reasoning from segmentation.
  arXiv  Detail & Related papers  (2025-08-11T16:59:06Z)
• ProstaTD: Bridging Surgical Triplet from Classification to Fully Supervised Detection [54.270188252068145]
  ProstaTD is a large-scale dataset for surgical triplet detection developed from the technically demanding domain of robot-assisted prostatectomy.<n>The dataset comprises 71,775 video frames and 196,490 annotated triplet instances, collected from 21 surgeries performed across multiple institutions.<n>ProstaTD is the largest and most diverse surgical triplet dataset to date, moving the field from simple classification to full detection with precise spatial and temporal boundaries.
  arXiv  Detail & Related papers  (2025-06-01T19:29:39Z)
• Language-Driven Visual Consensus for Zero-Shot Semantic Segmentation [114.72734384299476]
  We propose a Language-Driven Visual Consensus (LDVC) approach, fostering improved alignment of semantic and visual information. We leverage class embeddings as anchors due to their discrete and abstract nature, steering vision features toward class embeddings. Our approach significantly boosts the capacity of segmentation models for unseen classes.
  arXiv  Detail & Related papers  (2024-03-13T11:23:55Z)
• Pixel-Wise Recognition for Holistic Surgical Scene Understanding [33.40319680006502]
  This paper presents the Holistic and Multi-Granular Surgical Scene Understanding of Prostatectomies dataset.<n>Our benchmark models surgical scene understanding as a hierarchy of complementary tasks with varying levels of granularity.<n>To exploit our proposed benchmark, we introduce the Transformers for Actions, Phases, Steps, and Instrument (TAPIS) model.
  arXiv  Detail & Related papers  (2024-01-20T09:09:52Z)
• Robust Medical Instrument Segmentation Challenge 2019 [56.148440125599905]
  Intraoperative tracking of laparoscopic instruments is often a prerequisite for computer and robotic-assisted interventions. Our challenge was based on a surgical data set comprising 10,040 annotated images acquired from a total of 30 surgical procedures. The results confirm the initial hypothesis, namely that algorithm performance degrades with an increasing domain gap.
  arXiv  Detail & Related papers  (2020-03-23T14:35:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.