MovieRecapsQA: A Multimodal Open-Ended Video Question-Answering Benchmark

• URL: http://arxiv.org/abs/2601.02536v1
• Date: Mon, 05 Jan 2026 20:17:25 GMT
• Title: MovieRecapsQA: A Multimodal Open-Ended Video Question-Answering Benchmark
• Authors: Shaden Shaar, Bradon Thymes, Sirawut Chaixanien, Claire Cardie, Bharath Hariharan,
• Abstract summary: MovieRecapsQA is an open-ended VideoQA benchmark that supplies explicit textual context of the input.<n>Our benchmark provides videos of multiple lengths (i.e., recap-segments, movie-segments) and categorizations of questions (by modality and type) to enable fine-grained analysis.
• Score: 32.452556002879255
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Understanding real-world videos such as movies requires integrating visual and dialogue cues to answer complex questions. Yet existing VideoQA benchmarks struggle to capture this multimodal reasoning and are largely not open-ended, given the difficulty of evaluating free-form answers. In this paper, we introduce a novel open-ended multi-modal VideoQA benchmark, MovieRecapsQA created using movie recap videos--a distinctive type of YouTube content that summarizes a film by presenting its key events through synchronized visual (recap video) and textual (recap summary) modalities. Using the recap summary, we generate $\approx 8.2$ K question-answer (QA) pairs (aligned with movie-subtitles) and provide the necessary "facts" needed to verify an answer in a reference-free manner. To our knowledge, this is the first open-ended VideoQA benchmark that supplies explicit textual context of the input (video and/or text); which we use for evaluation. Our benchmark provides videos of multiple lengths (i.e., recap-segments, movie-segments) and categorizations of questions (by modality and type) to enable fine-grained analysis. We evaluate the performance of seven state-of-the-art MLLMs using our benchmark and observe that: 1) visual-only questions remain the most challenging; 2) models default to textual inputs whenever available; 3) extracting factually accurate information from video content is still difficult for all models; and 4) proprietary and open-source models perform comparably on video-dependent questions.

Related papers

• SAMA: Towards Multi-Turn Referential Grounded Video Chat with Large Language Models [93.73583158211115]
  Achieving fine-grained-temporal understanding in videos remains a major challenge for current Video Large Multimodels (Video LMMs)<n>We contribute in three core aspects: dataset, model, and benchmark.<n>First, we introduce SAMA-239K, a large-scale dataset comprising 15K videos specifically to enable joint learning of video understanding, grounding, and multi-turn video chat.<n>Second, we propose the SAMA model, which incorporates a versatile-temporal context aggregator and a Segment Model to jointly enhance fine-grained video comprehension and precise grounding capabilities.
  arXiv  Detail & Related papers  (2025-05-24T18:13:16Z)
• CaReBench: A Fine-Grained Benchmark for Video Captioning and Retrieval [24.203328970223527]
  We present CaReBench, a testing benchmark for fine-grained video captioning and retrieval.<n>Uniquely, it provides manually separated spatial annotations and temporal annotations for each video.<n>Based on this design, we introduce two evaluation metrics, ReBias and CapST, specifically tailored for video retrieval and video captioning tasks.
  arXiv  Detail & Related papers  (2024-12-31T15:53:50Z)
• Lost in Time: A New Temporal Benchmark for VideoLLMs [48.71203934876828]
  We show that the currently most used video-language benchmarks can be solved without requiring much temporal reasoning.<n>We propose TVBench, a novel open-source video multiple-choice question-answering benchmark.
  arXiv  Detail & Related papers  (2024-10-10T09:28:36Z)
• VideoQA in the Era of LLMs: An Empirical Study [108.37456450182054]
  Video Large Language Models (Video-LLMs) are flourishing and has advanced many video-intuitive tasks.<n>This work conducts a timely and comprehensive study of Video-LLMs' behavior in VideoQA.<n>Our analyses demonstrate that Video-LLMs excel in VideoQA; they can correlate contextual cues and generate plausible responses to questions about varied video contents.<n>However, models falter in handling video temporality, both in reasoning about temporal content ordering and grounding QA-relevant temporal moments.
  arXiv  Detail & Related papers  (2024-08-08T05:14:07Z)
• Needle In A Video Haystack: A Scalable Synthetic Evaluator for Video MLLMs [20.168429351519055]
  Video understanding is a crucial next step for multimodal large language models (LMLMs)<n>We propose VideoNIAH (Video Needle In A Haystack), a benchmark construction framework through synthetic video generation.<n>We conduct a comprehensive evaluation of both proprietary and open-source models, uncovering significant differences in their video understanding capabilities.
  arXiv  Detail & Related papers  (2024-06-13T17:50:05Z)
• FunQA: Towards Surprising Video Comprehension [64.58663825184958]
  We introduce FunQA, a challenging video question-answering dataset. FunQA covers three previously unexplored types of surprising videos: HumorQA, CreativeQA, and MagicQA. In total, the FunQA benchmark consists of 312K free-text QA pairs derived from 4.3K video clips.
  arXiv  Detail & Related papers  (2023-06-26T17:59:55Z)
• Video Question Answering with Iterative Video-Text Co-Tokenization [77.66445727743508]
  We propose a novel multi-stream video encoder for video question answering. We experimentally evaluate the model on several datasets, such as MSRVTT-QA, MSVD-QA, IVQA. Our model reduces the required GFLOPs from 150-360 to only 67, producing a highly efficient video question answering model.
  arXiv  Detail & Related papers  (2022-08-01T15:35:38Z)
• Fill-in-the-blank as a Challenging Video Understanding Evaluation Framework [19.031957183047048]
  We introduce a novel dataset consisting of 28,000 videos and fill-in-the-blank tests. We show that both a multimodal model and a strong language model have a large gap with human performance.
  arXiv  Detail & Related papers  (2021-04-09T04:00:10Z)
• Dense-Caption Matching and Frame-Selection Gating for Temporal Localization in VideoQA [96.10612095576333]
  We propose a video question answering model which effectively integrates multi-modal input sources and finds the temporally relevant information to answer questions. Our model is also comprised of dual-level attention (word/object and frame level), multi-head self-cross-integration for different sources (video and dense captions), and which pass more relevant information to gates. We evaluate our model on the challenging TVQA dataset, where each of our model components provides significant gains, and our overall model outperforms the state-of-the-art by a large margin.
  arXiv  Detail & Related papers  (2020-05-13T16:35:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.