CourseTimeQA: A Lecture-Video Benchmark and a Latency-Constrained Cross-Modal Fusion Method for Timestamped QA

• URL: http://arxiv.org/abs/2512.00360v1
• Date: Sat, 29 Nov 2025 07:06:51 GMT
• Title: CourseTimeQA: A Lecture-Video Benchmark and a Latency-Constrained Cross-Modal Fusion Method for Timestamped QA
• Abstract summary: Given a natural-language query, the system retrieves relevant timestamped segments and synthesizes a grounded answer.<n>We present CourseTimeQA (52.3 h, 902 queries across six courses) and a lightweight, latency-constrained cross-modal retriever (CrossFusion-RAG)<n>On CourseTimeQA, CrossFusion-RAG improves nDCG@10 by 0.10 and MRR by 0.08 over a strong BLIP-2 retriever while achieving approximately 1.55 s median end-to-end latency on a single A100.
• Score: 3.0867286767516937
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We study timestamped question answering over educational lecture videos under a single-GPU latency/memory budget. Given a natural-language query, the system retrieves relevant timestamped segments and synthesizes a grounded answer. We present CourseTimeQA (52.3 h, 902 queries across six courses) and a lightweight, latency-constrained cross-modal retriever (CrossFusion-RAG) that combines frozen encoders, a learned 512->768 vision projection, shallow query-agnostic cross-attention over ASR and frames with a temporal-consistency regularizer, and a small cross-attentive reranker. On CourseTimeQA, CrossFusion-RAG improves nDCG@10 by 0.10 and MRR by 0.08 over a strong BLIP-2 retriever while achieving approximately 1.55 s median end-to-end latency on a single A100. Closest comparators (zero-shot CLIP multi-frame pooling; CLIP + cross-encoder reranker + MMR; learned late-fusion gating; text-only hybrid with cross-encoder reranking and its MMR variant; caption-augmented text retrieval; non-learned temporal smoothing) are evaluated under matched hardware and indexing. We report robustness across ASR noise (WER quartiles), diagnostics for temporal localization, and full training/tuning details to support reproducible comparison.

Related papers

• Intrinsic Temporal Adaptation of CLIP for Partially Relevant Video Retrieval [40.983482307220676]
  PRVR aims to retrieve untrimmed videos that contain moments relevant to a text query.<n>We propose an Intrinsic Temporal Adaptation (ITA) framework for PRVR.<n>Our method achieves state-of-the-art performance on PRVR benchmarks.
  arXiv  Detail & Related papers  (2026-09-04T06:58:59Z)
• Audio-Text Cross-Attention with Psycholinguistic Support Features for Ambivalence/Hesitancy Recognition [0.6042845803090501]
  We present an audio-text system for the Ambivalence/Hesitancy Video Recognition Challenge of the 11th ABAW Competition.<n>The method excludes visual frames and represents each video as overlapping 5-second aligned with transcript timestamps.<n>Each window combines a 320-dimensional prosodic audio descriptor, a 768-dimensional emotion-oriented RoBERTa embedding, and 74 handcrafted features capturing uncertainty, hedging, and attitudinal conflict.
  arXiv  Detail & Related papers  (2026-07-15T00:11:24Z)
• TempRet: Temporal Enhancement and Two-Stage Reranking for CVPR 2026 EPIC-KITCHENS-100 Multi-Instance Retrieval Challenge [71.10535279591527]
  We present our solution, termed TempRet, to the CVPR 2026 EPIC-KITCHENS-100 MIR challenge.<n>Our approach builds upon a CLIP-based dual-encoder backbone and introduces two key components to address the temporal and cross-modal challenges.<n>Our method achieves 67.97% average mAP and 82.92% average nDCG on the EK-100 MIR benchmark.
  arXiv  Detail & Related papers  (2026-05-23T08:37:39Z)
• LE-NeuS: Latency-Efficient Neuro-Symbolic Video Understanding via Adaptive Temporal Verification [14.954035477725276]
  We present LE-NeuS, a latency-efficient neuro-symbolic framework that preserves the accuracy benefits of temporal logic-guided video understanding.<n>On LongVideoBench and Video-MME benchmarks, LE-NeuS reduces the latency gap from 90x to approximately 10x while maintaining >10% accuracy gains on temporally complex queries.
  arXiv  Detail & Related papers  (2026-02-26T23:28:13Z)
• Exploring Multimodal LMMs for Online Episodic Memory Question Answering on the Edge [9.701124246177661]
  We investigate the feasibility of using Multimodal Large Language Models (MLLMs) for real-time online episodic memory question answering.<n>Experiments on the QAEgo4D-Closed benchmark analyze the performance of Multimodal Large Language Models (MLLMs) within strict resource boundaries.
  arXiv  Detail & Related papers  (2026-02-25T22:30:56Z)
• PAS: A Training-Free Stabilizer for Temporal Encoding in Video LLMs [57.790910044227935]
  Video LLMs suffer from temporal inconsistency: small shifts in frame timing can flip attention and suppress relevant frames.<n>We present Phase Aggregated Smoothing (PAS), a training-free mechanism that applies small opposed phase offsets across heads and then aggregates their outputs.<n>Our analysis shows that the RoPE rotated logit can be approximated as a content dot product scaled by a time kernel; smoothing this kernel yields Lipschitz stability of attention to small temporal shifts; multi phase averaging attenuates high frequency ripples while preserving per-head spectra under Nyquist-valid sampling.
  arXiv  Detail & Related papers  (2025-11-14T05:56:47Z)
• Whisfusion: Parallel ASR Decoding via a Diffusion Transformer [7.327454599174306]
  Whisfusion is a framework to fuse a pre-trained Whisper encoder with a text diffusion decoder.<n>A lightweight cross-attention adapter trained via parameter-efficient fine-tuning (PEFT) bridges the two modalities.<n>Fine-tuned solely on LibriSpeech (960h), Whisfusion achieves a lower WER than Whisper-tiny, and offers comparable latency on short audio.
  arXiv  Detail & Related papers  (2025-08-09T17:20:54Z)
• Learning Frequency and Memory-Aware Prompts for Multi-Modal Object Tracking [74.15663758681849]
  We present Learning Frequency and Memory-Aware Prompts, a dual-adapter framework that injects lightweight prompts into a frozen RGB tracker.<n>A frequency-guided visual adapter adaptively transfers complementary cues across modalities.<n>A multilevel memory adapter with short, long, and permanent memory stores, updates, and retrieves reliable temporal context.
  arXiv  Detail & Related papers  (2025-06-30T15:38:26Z)
• From Play to Replay: Composed Video Retrieval for Temporally Fine-Grained Videos [48.666667545084835]
  Composed Video Retrieval (CoVR) retrieves a target video given a query video and a modification text describing the intended change.<n>We introduce TF-CoVR, the first large-scale benchmark dedicated to temporally fine-grained CoVR.<n> TF-CoVR focuses on gymnastics and diving and provides 180K triplets drawn from FineGym and FineDiving.
  arXiv  Detail & Related papers  (2025-06-05T17:31:17Z)
• RTime-QA: A Benchmark for Atomic Temporal Event Understanding in Large Multi-modal Models [85.59909303288921]
  We introduce RTime-QA, a novel benchmark designed to assess the atomic temporal event understanding ability of Large Multi-modal Models (LMMs)<n>RTime-QA comprises 822 high-quality, carefully-curated video-text questions, each meticulously annotated by human experts.<n>To advance LMMs' temporal event understanding ability, we further introduce RTime-IT, a 14k instruction-tuning dataset that employs a similar annotation process as RTime-QA.
  arXiv  Detail & Related papers  (2025-05-25T12:44:12Z)
• A Flexible Recursive Network for Video Stereo Matching Based on Residual Estimation [0.9362376508480733]
  RecSM is a network based on residual estimation for video stereo matching. With a stack count of 3, RecSM achieves a 4x speed improvement compared to ACVNet, running at 0.054 seconds based on one NVIDIA 2080TI GPU.
  arXiv  Detail & Related papers  (2024-06-05T14:49:14Z)
• Temporally Consistent Referring Video Object Segmentation with Hybrid Memory [98.80249255577304]
  We propose an end-to-end R-VOS paradigm that explicitly models temporal consistency alongside the referring segmentation. Features of frames with automatically generated high-quality reference masks are propagated to segment remaining frames. Extensive experiments demonstrate that our approach enhances temporal consistency by a significant margin.
  arXiv  Detail & Related papers  (2024-03-28T13:32:49Z)
• Learning for Video Compression with Recurrent Auto-Encoder and Recurrent Probability Model [164.7489982837475]
  This paper proposes a Recurrent Learned Video Compression (RLVC) approach with the Recurrent Auto-Encoder (RAE) and Recurrent Probability Model ( RPM) The RAE employs recurrent cells in both the encoder and decoder to exploit the temporal correlation among video frames. Our approach achieves the state-of-the-art learned video compression performance in terms of both PSNR and MS-SSIM.
  arXiv  Detail & Related papers  (2020-06-24T08:46:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.