SurgViVQA: Temporally-Grounded Video Question Answering for Surgical Scene Understanding

• URL: http://arxiv.org/abs/2511.03325v2
• Date: Thu, 06 Nov 2025 17:28:59 GMT
• Title: SurgViVQA: Temporally-Grounded Video Question Answering for Surgical Scene Understanding
• Abstract summary: Video Question Answering (VideoQA) in the surgical domain aims to enhance intraoperative understanding by enabling AI models to reason over temporally coherent events.<n>We propose SurgViVQA, a model that extends visual reasoning from static images to dynamic surgical scenes.<n>It uses a Masked Video-Text to fuse video and question features, capturing temporal cues such as motion and tool--tissue interactions.
• Score: 11.424693319143715
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Video Question Answering (VideoQA) in the surgical domain aims to enhance intraoperative understanding by enabling AI models to reason over temporally coherent events rather than isolated frames. Current approaches are limited to static image features, and available datasets often lack temporal annotations, ignoring the dynamics critical for accurate procedural interpretation. We propose SurgViVQA, a surgical VideoQA model that extends visual reasoning from static images to dynamic surgical scenes. It uses a Masked Video--Text Encoder to fuse video and question features, capturing temporal cues such as motion and tool--tissue interactions, which a fine-tuned large language model (LLM) then decodes into coherent answers. To evaluate its performance, we curated REAL-Colon-VQA, a colonoscopic video dataset that includes motion-related questions and diagnostic attributes, as well as out-of-template questions with rephrased or semantically altered formulations to assess model robustness. Experimental validation on REAL-Colon-VQA and the public EndoVis18-VQA dataset shows that SurgViVQA outperforms existing image-based VQA benchmark models, particularly in keyword accuracy, improving over PitVQA by +11\% on REAL-Colon-VQA and +9\% on EndoVis18-VQA. A perturbation study on the questions further confirms improved generalizability and robustness to variations in question phrasing. SurgViVQA and the REAL-Colon-VQA dataset provide a framework for temporally-aware understanding in surgical VideoQA, enabling AI models to interpret dynamic procedural contexts more effectively. Code and dataset available at https://github.com/madratak/SurgViVQA.

Related papers

• SurgTEMP: Temporal-Aware Surgical Video Question Answering with Text-guided Visual Memory for Laparoscopic Cholecystectomy [12.553409376394162]
  SurgTEMP is a multimodal visual memory framework for surgical video question answering.<n>We introduce CholeVidQA-32K, a surgical video question answering dataset comprising 32K open-ended QA pairs and 3,855 video segments.<n>In comprehensive evaluations against state-of-the-art open-source multimodal and video LLMs (fine-tuned and zero-shot), SurgTEMP achieves substantial performance improvements.
  arXiv  Detail & Related papers  (2026-03-31T16:27:46Z)
• ReAG: Reasoning-Augmented Generation for Knowledge-based Visual Question Answering [54.72902502486611]
  ReAG is a Reasoning-Augmented Multimodal RAG approach that combines coarse- and fine-grained retrieval with a critic model that filters irrelevant passages.<n>ReAG significantly outperforms prior methods, improving answer accuracy and providing interpretable reasoning grounded in retrieved evidence.
  arXiv  Detail & Related papers  (2025-11-27T19:01:02Z)
• SurgAnt-ViVQA: Learning to Anticipate Surgical Events through GRU-Driven Temporal Cross-Attention [10.149538951173598]
  Anticipating forthcoming surgical events is vital for real-time assistance in endonasal transsphenoidal pituitary surgery.<n>Most visual question answering (VQA) systems reason on isolated frames with static vision language alignment.<n>We introduce PitVQA-Anticipation, the first VQA dataset designed for forward looking surgical reasoning.
  arXiv  Detail & Related papers  (2025-11-05T04:55:11Z)
• Towards Fine-Grained Video Question Answering [17.582244704442747]
  This paper introduces the Multi-Object Multi-Actor Question Answering (MOMA-QA) dataset.<n>With ground truth scene graphs and temporal interval annotations, MOMA-QA is ideal for developing models for fine-grained video understanding.<n>We present a novel video-language model, SGVLM, which incorporates a scene graph predictor, an efficient frame retriever, and a pre-trained large language model for temporal localization and fine-grained relationship understanding.
  arXiv  Detail & Related papers  (2025-03-10T01:02:01Z)
• Cross-modal Causal Relation Alignment for Video Question Grounding [44.97933293141372]
  Video question grounding (VideoQG) requires models to answer the questions and simultaneously infer the relevant video segments to support the answers.<n>Existing VideoQG methods usually suffer from spurious cross-modal correlations, leading to a failure to identify the dominant visual scenes that align with the intended question.<n>We propose a novel VideoQG framework named Cross-modal Causal Relation Alignment (CRA), to eliminate spurious correlations and improve the causal consistency between question-answering and video temporal grounding.
  arXiv  Detail & Related papers  (2025-03-05T01:36:32Z)
• TimeLogic: A Temporal Logic Benchmark for Video QA [64.32208175236323]
  We introduce the TimeLogic QA (TLQA) framework to automatically generate temporal logical questions.<n>We leverage 4 datasets, STAR, Breakfast, AGQA, and CrossTask, and generate 2k and 10k QA pairs for each category.<n>We assess the VideoQA model's temporal reasoning performance on 16 categories of temporal logic with varying temporal complexity.
  arXiv  Detail & Related papers  (2025-01-13T11:12:59Z)
• Advancing Surgical VQA with Scene Graph Knowledge [45.05847978115387]
  We aim to advance Visual Question Answering in the surgical context with scene graph knowledge. We build surgical scene graphs using spatial and action information of instruments and anatomies. We propose SSG-QA-Net, a novel surgical VQA model incorporating a lightweight Scene-embedded Interaction Module (SIM)
  arXiv  Detail & Related papers  (2023-12-15T22:50:12Z)
• Surgical-VQLA: Transformer with Gated Vision-Language Embedding for Visual Question Localized-Answering in Robotic Surgery [18.248882845789353]
  We develop a surgical question-answering system to facilitate robot-assisted surgical scene and activity understanding from recorded videos. Most of the existing VQA methods require an object detector and regions based feature extractor to extract visual features and fuse them with the embedded text of the question for answer generation. We propose Visual Question Localized-Answering in Robotic Surgery (Surgical-VQLA) to localize the specific surgical area during the answer prediction.
  arXiv  Detail & Related papers  (2023-05-19T14:13:47Z)
• Frame-Subtitle Self-Supervision for Multi-Modal Video Question Answering [73.11017833431313]
  Multi-modal video question answering aims to predict correct answer and localize the temporal boundary relevant to the question. We devise a weakly supervised question grounding (WSQG) setting, where only QA annotations are used. We transform the correspondence between frames and subtitles to Frame-Subtitle (FS) self-supervision, which helps to optimize the temporal attention scores.
  arXiv  Detail & Related papers  (2022-09-08T07:20:51Z)
• Invariant Grounding for Video Question Answering [72.87173324555846]
  Video Question Answering (VideoQA) is the task of answering questions about a video. In leading VideoQA models, the typical learning objective, empirical risk minimization (ERM), latches on superficial correlations between video-question pairs and answers. We propose a new learning framework, Invariant Grounding for VideoQA (IGV), to ground the question-critical scene.
  arXiv  Detail & Related papers  (2022-06-06T04:37:52Z)
• Human-Adversarial Visual Question Answering [62.30715496829321]
  We benchmark state-of-the-art VQA models against human-adversarial examples. We find that a wide range of state-of-the-art models perform poorly when evaluated on these examples.
  arXiv  Detail & Related papers  (2021-06-04T06:25:32Z)
• HySTER: A Hybrid Spatio-Temporal Event Reasoner [75.41988728376081]
  We present the HySTER: a Hybrid Spatio-Temporal Event Reasoner to reason over physical events in videos. We define a method based on general temporal, causal and physics rules which can be transferred across tasks. This work sets the foundations for the incorporation of inductive logic programming in the field of VideoQA.
  arXiv  Detail & Related papers  (2021-01-17T11:07:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.