RoboSurg-VQA: A Multimodal Benchmark for Surgical Segmentation-Aware Visual Question Answering

• URL: http://arxiv.org/abs/2605.23068v1
• Date: Thu, 21 May 2026 21:58:11 GMT
• Title: RoboSurg-VQA: A Multimodal Benchmark for Surgical Segmentation-Aware Visual Question Answering
• Abstract summary: We present textbfRoboSurg-VQA, a segmentation-aware visual question answering (VQA) benchmark.<n>Each frame is paired with a fixed set of clinically motivated questions spanning procedure context, anatomy (including region), imaging modality/view, surgical artefacts, image quality, and basic visibility and spatial attributes.<n>We report benchmark statistics, sanity baselines, and common evaluation challenges under challenging surgical conditions.
• Score: 15.020056500396478
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Reliable visual understanding in robot-assisted and minimally invasive surgery (RMIS/MIS) demands more than accurate masks: in clinical practice, clinicians pose language-like questions about procedural context, visibility, artefacts, and the presence of anatomical structures and surgical instruments, often under degraded views caused by occlusion, smoke, bleeding, and specular highlights. We present \textbf{RoboSurg-VQA}, a segmentation-aware visual question answering (VQA) benchmark built by repurposing public surgical segmentation datasets under a shared schema. Each frame is paired with a fixed set of clinically motivated questions spanning procedure context, anatomy (including region), imaging modality/view, surgical artefacts, image quality, and basic visibility and spatial attributes, with closed answer sets to enable consistent evaluation. To scale annotation, we generate candidate answers via constrained prompting with automatic validity and consistency checks, followed by human auditing to improve plausibility and label consistency. We report benchmark statistics, sanity baselines, and common evaluation challenges under challenging surgical conditions. The code will be available on https://github.com/ziyangwang007/Robosurg-VQA.

Related papers

• Object Tokens as a Bridge Between Segmentation and Visual Question Answering in Robotic Surgery [1.4146817306459933]
  We propose a unified framework that jointly performs pixel-level segmentation and visual question answering within a single framework.<n>We evaluate the proposed method on the private RAMIE (Robot-Assisted Minimally Invasive Esophagectomy) dataset and the public EndoVis18 dataset.<n>Results demonstrate that incorporating context-aware object tokens into vision-language models improves fine-grained surgical scene understanding.
  arXiv  Detail & Related papers  (2026-06-14T15:30:16Z)
• SurgCheck: Do Vision-Language Models Really Look at Images in Surgical VQA? [4.831595834944456]
  Vision-language models (VLMs) have shown promising performance in surgical visual question answering (VQA)<n>It remains unclear whether reported performance reflects visual understanding or reliance on such linguistic shortcuts.<n>We introduce SurgCheck, a diagnostic benchmark for quantifying linguistic shortcut reliance in surgical VQA.
  arXiv  Detail & Related papers  (2026-05-03T14:47:03Z)
• SurgTEMP: Temporal-Aware Surgical Video Question Answering with Text-guided Visual Memory for Laparoscopic Cholecystectomy [12.553409376394162]
  SurgTEMP is a multimodal visual memory framework for surgical video question answering.<n>We introduce CholeVidQA-32K, a surgical video question answering dataset comprising 32K open-ended QA pairs and 3,855 video segments.<n>In comprehensive evaluations against state-of-the-art open-source multimodal and video LLMs (fine-tuned and zero-shot), SurgTEMP achieves substantial performance improvements.
  arXiv  Detail & Related papers  (2026-03-31T16:27:46Z)
• GroundedSurg: A Multi-Procedure Benchmark for Language-Conditioned Surgical Tool Segmentation [1.9981885081131854]
  We introduce GroundedSurg, the first language-conditioned, instance-level surgical grounding benchmark.<n>The dataset spans ophthalmic, laparoscopic, robotic, and open procedures, encompassing diverse instrument types, imaging conditions, and operative complexities.
  arXiv  Detail & Related papers  (2026-03-01T13:49:53Z)
• SurgVLM: A Large Vision-Language Model and Systematic Evaluation Benchmark for Surgical Intelligence [72.10889173696928]
  We propose SurgVLM, one of the first large vision-language foundation models for surgical intelligence.<n>We construct a large-scale multimodal surgical database, SurgVLM-DB, spanning more than 16 surgical types and 18 anatomical structures.<n>Building upon this comprehensive dataset, we propose SurgVLM, which is built upon Qwen2.5-VL, and undergoes instruction tuning to 10+ surgical tasks.
  arXiv  Detail & Related papers  (2025-06-03T07:44:41Z)
• ProstaTD: Bridging Surgical Triplet from Classification to Fully Supervised Detection [54.270188252068145]
  ProstaTD is a large-scale dataset for surgical triplet detection developed from the technically demanding domain of robot-assisted prostatectomy.<n>The dataset comprises 71,775 video frames and 196,490 annotated triplet instances, collected from 21 surgeries performed across multiple institutions.<n>ProstaTD is the largest and most diverse surgical triplet dataset to date, moving the field from simple classification to full detection with precise spatial and temporal boundaries.
  arXiv  Detail & Related papers  (2025-06-01T19:29:39Z)
• Benchmarking Laparoscopic Surgical Image Restoration and Beyond [54.28852320829451]
  In laparoscopic surgery, a clear and high-quality visual field is critical for surgeons to make accurate decisions.<n> persistent visual degradation, including smoke generated by energy devices, lens fogging from thermal gradients, and lens contamination pose risks to patient safety.<n>We introduce a real-world open-source surgical image restoration dataset covering laparoscopic environments, called SurgClean.
  arXiv  Detail & Related papers  (2025-05-25T14:17:56Z)
• Surgical Foundation Model Leveraging Compression and Entropy Maximization for Image-Guided Surgical Assistance [50.486523249499115]
  Real-time video understanding is critical to guide procedures in minimally invasive surgery (MIS)<n>We propose Compress-to-Explore (C2E), a novel self-supervised framework to learn compact, informative representations from surgical videos.<n>C2E uses entropy-maximizing decoders to compress images while preserving clinically relevant details, improving encoder performance without labeled data.
  arXiv  Detail & Related papers  (2025-05-16T14:02:24Z)
• Surgeons vs. Computer Vision: A comparative analysis on surgical phase recognition capabilities [65.66373425605278]
  Automated Surgical Phase Recognition (SPR) uses Artificial Intelligence (AI) to segment the surgical workflow into its key events.<n>Previous research has focused on short and linear surgical procedures and has not explored if temporal context influences experts' ability to better classify surgical phases.<n>This research addresses these gaps, focusing on Robot-Assisted Partial Nephrectomy (RAPN) as a highly non-linear procedure.
  arXiv  Detail & Related papers  (2025-04-26T15:37:22Z)
• Advancing Surgical VQA with Scene Graph Knowledge [45.05847978115387]
  We aim to advance Visual Question Answering in the surgical context with scene graph knowledge. We build surgical scene graphs using spatial and action information of instruments and anatomies. We propose SSG-QA-Net, a novel surgical VQA model incorporating a lightweight Scene-embedded Interaction Module (SIM)
  arXiv  Detail & Related papers  (2023-12-15T22:50:12Z)
• Surgical-VQLA: Transformer with Gated Vision-Language Embedding for Visual Question Localized-Answering in Robotic Surgery [18.248882845789353]
  We develop a surgical question-answering system to facilitate robot-assisted surgical scene and activity understanding from recorded videos. Most of the existing VQA methods require an object detector and regions based feature extractor to extract visual features and fuse them with the embedded text of the question for answer generation. We propose Visual Question Localized-Answering in Robotic Surgery (Surgical-VQLA) to localize the specific surgical area during the answer prediction.
  arXiv  Detail & Related papers  (2023-05-19T14:13:47Z)
• Robust Medical Instrument Segmentation Challenge 2019 [56.148440125599905]
  Intraoperative tracking of laparoscopic instruments is often a prerequisite for computer and robotic-assisted interventions. Our challenge was based on a surgical data set comprising 10,040 annotated images acquired from a total of 30 surgical procedures. The results confirm the initial hypothesis, namely that algorithm performance degrades with an increasing domain gap.
  arXiv  Detail & Related papers  (2020-03-23T14:35:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.