SurgCheck: Do Vision-Language Models Really Look at Images in Surgical VQA?

• URL: http://arxiv.org/abs/2605.01911v2
• Date: Tue, 05 May 2026 13:32:54 GMT
• Title: SurgCheck: Do Vision-Language Models Really Look at Images in Surgical VQA?
• Abstract summary: Vision-language models (VLMs) have shown promising performance in surgical visual question answering (VQA)<n>It remains unclear whether reported performance reflects visual understanding or reliance on such linguistic shortcuts.<n>We introduce SurgCheck, a diagnostic benchmark for quantifying linguistic shortcut reliance in surgical VQA.
• Score: 4.831595834944456
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Purpose: Vision-language models (VLMs) have shown promising performance in surgical visual question answering (VQA). However, existing surgical VQA datasets often contain linguistic shortcuts, where question phrasing implicitly constrains the answer space. It remains unclear whether reported performance reflects visual understanding or reliance on such linguistic shortcuts. Methods: We introduce SurgCheck, a diagnostic benchmark for quantifying linguistic shortcut reliance in surgical VQA. SurgCheck employs a paired-question design in which each surgical frame is associated with an original question containing entity names and a less-biased counterpart that removes these names while preserving identical visual content and ground-truth answers. The resulting performance gap provides a diagnostic signal of shortcut reliance. To ensure that the less-biased question remains well-defined even without entity names, four grounding cues are incorporated: bounding box, arrow, spatial position, and periphrasis. We evaluate both general-purpose and surgical-specific VLMs under zero-shot and fine-tuned settings on SurgCheck. To evaluate open-ended zero-shot responses, we introduce an LLM-as-a-judge evaluation protocol. Results: Using SurgCheck, we observe consistent performance degradation on less-biased questions across five VLMs, despite identical visual inputs. Text-only ablation reveals minimal performance drops for action and target prediction, indicating that action and target prediction is largely driven by linguistic shortcuts rather than visual reasoning. Conclusion: SurgCheck provides a controlled diagnostic framework that exposes failure modes masked by linguistic bias in existing surgical VQA benchmarks. Our findings demonstrate that strong benchmark performance does not necessarily imply faithful visual understanding, underscoring the need for bias-aware evaluation in surgical VQA.

Related papers

• Object Tokens as a Bridge Between Segmentation and Visual Question Answering in Robotic Surgery [1.4146817306459933]
  We propose a unified framework that jointly performs pixel-level segmentation and visual question answering within a single framework.<n>We evaluate the proposed method on the private RAMIE (Robot-Assisted Minimally Invasive Esophagectomy) dataset and the public EndoVis18 dataset.<n>Results demonstrate that incorporating context-aware object tokens into vision-language models improves fine-grained surgical scene understanding.
  arXiv  Detail & Related papers  (2026-06-14T15:30:16Z)
• RoboSurg-VQA: A Multimodal Benchmark for Surgical Segmentation-Aware Visual Question Answering [15.020056500396478]
  We present textbfRoboSurg-VQA, a segmentation-aware visual question answering (VQA) benchmark.<n>Each frame is paired with a fixed set of clinically motivated questions spanning procedure context, anatomy (including region), imaging modality/view, surgical artefacts, image quality, and basic visibility and spatial attributes.<n>We report benchmark statistics, sanity baselines, and common evaluation challenges under challenging surgical conditions.
  arXiv  Detail & Related papers  (2026-05-21T21:58:11Z)
• MedObvious: Exposing the Medical Moravec's Paradox in VLMs via Clinical Triage [20.835664121303534]
  Vision Language Models (VLMs) are increasingly used for tasks like medical report generation and visual question answering.<n>In clinical practice, interpretation begins with pre-diagnostic sanity checks.<n>Existing benchmarks largely assume this step is solved, and therefore miss a critical failure mode.<n>We introduce MedObvious, a 1,880-task benchmark that isolates input validation as a set-level consistency capability.
  arXiv  Detail & Related papers  (2026-03-24T17:59:54Z)
• Seeing Through Words: Controlling Visual Retrieval Quality with Language Models [68.49490036960559]
  We propose a new paradigm of quality-controllable retrieval, which enriches short queries with contextual details while incorporating explicit notions of image quality.<n>Our key idea is to leverage a generative language model as a query completion function, extending underspecified queries into descriptive forms.<n>Our proposed approach significantly improves retrieval results and provides effective quality control, bridging the gap between the expressive capacity of modern VLMs and the underspecified nature of short user queries.
  arXiv  Detail & Related papers  (2026-02-24T18:20:57Z)
• SurgViVQA: Temporally-Grounded Video Question Answering for Surgical Scene Understanding [11.424693319143715]
  Video Question Answering (VideoQA) in the surgical domain aims to enhance intraoperative understanding by enabling AI models to reason over temporally coherent events.<n>We propose SurgViVQA, a model that extends visual reasoning from static images to dynamic surgical scenes.<n>It uses a Masked Video-Text to fuse video and question features, capturing temporal cues such as motion and tool--tissue interactions.
  arXiv  Detail & Related papers  (2025-11-05T09:40:16Z)
• SurgAnt-ViVQA: Learning to Anticipate Surgical Events through GRU-Driven Temporal Cross-Attention [10.149538951173598]
  Anticipating forthcoming surgical events is vital for real-time assistance in endonasal transsphenoidal pituitary surgery.<n>Most visual question answering (VQA) systems reason on isolated frames with static vision language alignment.<n>We introduce PitVQA-Anticipation, the first VQA dataset designed for forward looking surgical reasoning.
  arXiv  Detail & Related papers  (2025-11-05T04:55:11Z)
• S-Chain: Structured Visual Chain-of-Thought For Medicine [81.97605645734741]
  We introduce S-Chain, the first large-scale dataset of 12,000 expert-annotated medical images with bounding boxes and structured visual CoT (SV-CoT)<n>The dataset further supports 16 languages, totaling over 700k VQA pairs for broad multilingual applicability.<n>S-Chain establishes a new benchmark for grounded medical reasoning and paves the way toward more trustworthy and explainable medical vision-language models.
  arXiv  Detail & Related papers  (2025-10-26T15:57:14Z)
• Faithfulness-Aware Uncertainty Quantification for Fact-Checking the Output of Retrieval Augmented Generation [108.13261761812517]
  We introduce FRANQ (Faithfulness-based Retrieval Augmented UNcertainty Quantification), a novel method for hallucination detection in RAG outputs.<n>We present a new long-form Question Answering (QA) dataset annotated for both factuality and faithfulness.
  arXiv  Detail & Related papers  (2025-05-27T11:56:59Z)
• Trust but Verify: Programmatic VLM Evaluation in the Wild [62.14071929143684]
  Programmatic VLM Evaluation (PROVE) is a new benchmarking paradigm for evaluating VLM responses to open-ended queries. We benchmark the helpfulness-truthfulness trade-offs of a range ofVLMs on PROVE, finding that very few are in-fact able to achieve a good balance between the two.
  arXiv  Detail & Related papers  (2024-10-17T01:19:18Z)
• Procedure-Aware Surgical Video-language Pretraining with Hierarchical Knowledge Augmentation [51.222684687924215]
  Surgical video-language pretraining faces unique challenges due to the knowledge domain gap and the scarcity of multi-modal data.<n>We propose a hierarchical knowledge augmentation approach and a novel Procedure-Encoded Surgical Knowledge-Augmented Video-Language Pretraining framework to tackle these issues.
  arXiv  Detail & Related papers  (2024-09-30T22:21:05Z)
• Advancing Surgical VQA with Scene Graph Knowledge [45.05847978115387]
  We aim to advance Visual Question Answering in the surgical context with scene graph knowledge. We build surgical scene graphs using spatial and action information of instruments and anatomies. We propose SSG-QA-Net, a novel surgical VQA model incorporating a lightweight Scene-embedded Interaction Module (SIM)
  arXiv  Detail & Related papers  (2023-12-15T22:50:12Z)
• Rephrase, Augment, Reason: Visual Grounding of Questions for Vision-Language Models [59.05769810380928]
  Rephrase, Augment and Reason (RepARe) is a gradient-free framework that extracts salient details about the image using the underlying vision-language model. We show that RepARe can result in a 3.85% (absolute) increase in zero-shot accuracy on VQAv2, 6.41%, and 7.94% points increase on A-OKVQA, and VizWiz respectively.
  arXiv  Detail & Related papers  (2023-10-09T16:57:57Z)
• Surgical-VQLA: Transformer with Gated Vision-Language Embedding for Visual Question Localized-Answering in Robotic Surgery [18.248882845789353]
  We develop a surgical question-answering system to facilitate robot-assisted surgical scene and activity understanding from recorded videos. Most of the existing VQA methods require an object detector and regions based feature extractor to extract visual features and fuse them with the embedded text of the question for answer generation. We propose Visual Question Localized-Answering in Robotic Surgery (Surgical-VQLA) to localize the specific surgical area during the answer prediction.
  arXiv  Detail & Related papers  (2023-05-19T14:13:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.