TemporalDoRA: Temporal PEFT for Robust Surgical Video Question Answering

• URL: http://arxiv.org/abs/2603.09696v1
• Date: Tue, 10 Mar 2026 14:03:42 GMT
• Title: TemporalDoRA: Temporal PEFT for Robust Surgical Video Question Answering
• Abstract summary: We introduce TemporalDoRA, a video-specific PEFT formulation that extends Weight-Decomposed Low-Rank Adaptation.<n>By mixing information across frames within the adaptation subspace, TemporalDoRA steers updates toward temporally consistent visual cues.
• Score: 12.77631688698326
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Surgical Video Question Answering (VideoQA) requires accurate temporal grounding while remaining robust to natural variation in how clinicians phrase questions, where linguistic bias can arise. Standard Parameter Efficient Fine Tuning (PEFT) methods adapt pretrained projections without explicitly modeling frame-to-frame interactions within the adaptation pathway, limiting their ability to exploit sparse temporal evidence. We introduce TemporalDoRA, a video-specific PEFT formulation that extends Weight-Decomposed Low-Rank Adaptation by (i) inserting lightweight temporal Multi-Head Attention (MHA) inside the low-rank bottleneck of the vision encoder and (ii) selectively applying weight decomposition only to the trainable low-rank branch rather than the full adapted weight. This design enables temporally-aware updates while preserving a frozen backbone and stable scaling. By mixing information across frames within the adaptation subspace, TemporalDoRA steers updates toward temporally consistent visual cues and improves robustness with minimal parameter overhead. To benchmark this setting, we present REAL-Colon-VQA, a colonoscopy VideoQA dataset with 6,424 clip--question pairs, including paired rephrased Out-of-Template questions to evaluate sensitivity to linguistic variation. TemporalDoRA improves Out-of-Template performance, and ablation studies confirm that temporal mixing inside the low-rank branch is the primary driver of these gains. We also validate on EndoVis18-VQA adapted to short clips and observe consistent improvements on the Out-of-Template split. Code and dataset available at~\href{https://anonymous.4open.science/r/TemporalDoRA-BFC8/}{Anonymous GitHub}.

Related papers

• Intrinsic Temporal Adaptation of CLIP for Partially Relevant Video Retrieval [40.983482307220676]
  PRVR aims to retrieve untrimmed videos that contain moments relevant to a text query.<n>We propose an Intrinsic Temporal Adaptation (ITA) framework for PRVR.<n>Our method achieves state-of-the-art performance on PRVR benchmarks.
  arXiv  Detail & Related papers  (2026-09-04T06:58:59Z)
• SurgLQA: Scalable Long-Horizon Surgical Video Question Answering [44.46045775428032]
  Surgical Video Question Answering (VideoQA) provides a promising paradigm for dynamic intraoperative interpretation.<n>Existing approaches are predominantly restricted to images or short clips.<n>We propose SurgLQA, a unified long-horizon VideoQA framework for scalable surgical reasoning.
  arXiv  Detail & Related papers  (2026-05-18T06:24:07Z)
• Relaxing Anchor-Frame Dominance for Mitigating Hallucinations in Video Large Language Models [22.554751069064906]
  Recent Video Large Language Models (Video-LLMs) have demonstrated strong capability in video understanding, yet they still suffer from hallucinations.<n>We investigate a decoder-side phenomenon in which the model exhibits a temporally imbalanced concentration pattern.<n>We propose Decoder-side Temporal Rebalancing (DTR), a training-free, layer-selective inference method that rebalances temporal evidence allocation in middle-to-late decoder layers.
  arXiv  Detail & Related papers  (2026-04-14T11:05:42Z)
• GateRA: Token-Aware Modulation for Parameter-Efficient Fine-Tuning [51.79350934271497]
  GateRA is a unified framework that introduces token-aware modulation to dynamically adjust the strength of PEFT updates.<n>By incorporating adaptive gating into standard PEFT branches, GateRA enables selective, token-level adaptation.<n> Experiments on multiple commonsense reasoning benchmarks demonstrate that GateRA consistently outperforms or matches prior PEFT methods.
  arXiv  Detail & Related papers  (2025-11-15T17:55:47Z)
• PAS: A Training-Free Stabilizer for Temporal Encoding in Video LLMs [57.790910044227935]
  Video LLMs suffer from temporal inconsistency: small shifts in frame timing can flip attention and suppress relevant frames.<n>We present Phase Aggregated Smoothing (PAS), a training-free mechanism that applies small opposed phase offsets across heads and then aggregates their outputs.<n>Our analysis shows that the RoPE rotated logit can be approximated as a content dot product scaled by a time kernel; smoothing this kernel yields Lipschitz stability of attention to small temporal shifts; multi phase averaging attenuates high frequency ripples while preserving per-head spectra under Nyquist-valid sampling.
  arXiv  Detail & Related papers  (2025-11-14T05:56:47Z)
• SurgViVQA: Temporally-Grounded Video Question Answering for Surgical Scene Understanding [11.424693319143715]
  Video Question Answering (VideoQA) in the surgical domain aims to enhance intraoperative understanding by enabling AI models to reason over temporally coherent events.<n>We propose SurgViVQA, a model that extends visual reasoning from static images to dynamic surgical scenes.<n>It uses a Masked Video-Text to fuse video and question features, capturing temporal cues such as motion and tool--tissue interactions.
  arXiv  Detail & Related papers  (2025-11-05T09:40:16Z)
• SurgAnt-ViVQA: Learning to Anticipate Surgical Events through GRU-Driven Temporal Cross-Attention [10.149538951173598]
  Anticipating forthcoming surgical events is vital for real-time assistance in endonasal transsphenoidal pituitary surgery.<n>Most visual question answering (VQA) systems reason on isolated frames with static vision language alignment.<n>We introduce PitVQA-Anticipation, the first VQA dataset designed for forward looking surgical reasoning.
  arXiv  Detail & Related papers  (2025-11-05T04:55:11Z)
• EntroPE: Entropy-Guided Dynamic Patch Encoder for Time Series Forecasting [50.794700596484894]
  We propose EntroPE (Entropy-Guided Dynamic Patch), a novel, temporally informed framework that dynamically detects transition points via conditional entropy.<n>This preserves temporal structure while retaining the computational benefits of patching.<n> Experiments across long-term forecasting benchmarks demonstrate that EntroPE improves both accuracy and efficiency.
  arXiv  Detail & Related papers  (2025-09-30T12:09:56Z)
• RAP: Efficient Text-Video Retrieval with Sparse-and-Correlated Adapter [77.0205013713008]
  Text-Video Retrieval (TVR) aims to align relevant video content with natural language queries. To date, most state-of-the-art TVR methods learn image-to-video transfer learning based on large-scale pre-trained vision models. We propose a sparse-andcorrelated AdaPter (RAP) to fine-tune the pre-trained model with a few parameterized layers.
  arXiv  Detail & Related papers  (2024-05-29T19:23:53Z)
• RT-GAN: Recurrent Temporal GAN for Adding Lightweight Temporal Consistency to Frame-Based Domain Translation Approaches [3.7873597471903944]
  We present a lightweight solution with a tunable temporal parameter, RT-GAN, for adding temporal consistency to individual frame-based approaches.<n>We demonstrate the effectiveness of our approach on two challenging use cases in colonoscopy.
  arXiv  Detail & Related papers  (2023-10-02T03:13:26Z)
• Transform-Equivariant Consistency Learning for Temporal Sentence Grounding [66.10949751429781]
  We introduce a novel Equivariant Consistency Regulation Learning framework to learn more discriminative representations for each video. Our motivation comes from that the temporal boundary of the query-guided activity should be consistently predicted. In particular, we devise a self-supervised consistency loss module to enhance the completeness and smoothness of the augmented video.
  arXiv  Detail & Related papers  (2023-05-06T19:29:28Z)
• Frame-Subtitle Self-Supervision for Multi-Modal Video Question Answering [73.11017833431313]
  Multi-modal video question answering aims to predict correct answer and localize the temporal boundary relevant to the question. We devise a weakly supervised question grounding (WSQG) setting, where only QA annotations are used. We transform the correspondence between frames and subtitles to Frame-Subtitle (FS) self-supervision, which helps to optimize the temporal attention scores.
  arXiv  Detail & Related papers  (2022-09-08T07:20:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.