Two Causally Related Needles in a Video Haystack

• URL: http://arxiv.org/abs/2505.19853v1
• Date: Mon, 26 May 2025 11:37:34 GMT
• Title: Two Causally Related Needles in a Video Haystack
• Abstract summary: We propose a benchmark that assesses the ability to extract information from two separate locations in a long video and understand them jointly.<n>Caul2Needles introduces 2-needle questions, which require extracting information from both the cause and effect human-behavior events in a long video.<n>Our experiments reveal that models excelling in pre-existing benchmarks struggle with 2-needle visual grounding.
• Score: 4.1753350239906295
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Evaluating the video understanding capabilities of Video-Language Models (VLMs) remains a significant challenge. We propose a long-context video understanding benchmark, Causal2Needles, that assesses two crucial abilities insufficiently evaluated by existing benchmarks: (1) the ability to extract information from two separate locations in a long video and understand them jointly, and (2) the ability to model the world in terms of cause and effect in human behaviors. Specifically, Causal2Needles introduces 2-needle questions, which require extracting information from both the cause and effect human-behavior events in a long video and the associated narration text. To prevent textual bias, these questions comprise two complementary formats: one asking to identify the video clip containing the answer, and one asking for the textual description of an unrelated visual detail from that video clip. Our experiments reveal that models excelling in pre-existing benchmarks struggle with 2-needle visual grounding, and the model performance is negatively correlated with the distance between the two needles. These findings highlight critical limitations in current VLMs.

Related papers

• CaST-Bench: Benchmarking Causal Chain-Grounded Spatio-Temporal Reasoning for Video Question Answering [27.144152466342103]
  Cause-and-effect reasoning in video is a significant challenge for Vision-Language Models (VLMs)<n>We introduce CaST-Bench, a benchmark for Causal-Temporal Video Video Reasoning SST-Bench.<n>We construct a high-quality dataset of 2,066 questions over 1,015 videos, with causal chains annotated by temporal segments and bounding-box tracks.
  arXiv  Detail & Related papers  (2026-05-22T04:19:29Z)
• VersaVid-R1: A Versatile Video Understanding and Reasoning Model from Question Answering to Captioning Tasks [44.30048178589923]
  We introduce two novel datasets designed to stimulate the model's advanced video understanding and reasoning abilities.<n>We develop VersaVid-R1, the first versatile video understanding and reasoning model under the Reason-Then-Respond paradigm.
  arXiv  Detail & Related papers  (2025-06-10T03:57:53Z)
• Movie Facts and Fibs (MF$^2$): A Benchmark for Long Movie Understanding [97.05584099530226]
  We introduce MF$2$, a new benchmark for evaluating whether models can comprehend, consolidate, and recall key narrative information from full-length movies.<n>For each pair, models must correctly identify both the true and false claims.<n>Our experiments demonstrate that both open-weight and closed state-of-the-art models fall well short of human performance.
  arXiv  Detail & Related papers  (2025-06-06T17:58:36Z)
• SAMA: Towards Multi-Turn Referential Grounded Video Chat with Large Language Models [80.3895950009792]
  Achieving fine-grained-temporal understanding in videos remains a major challenge for current Video Large Multimodels (Video LMMs)<n>We contribute in three core aspects: dataset, model, and benchmark.<n>First, we introduce SAMA-239K, a large-scale dataset comprising 15K videos specifically to enable joint learning of video understanding, grounding, and multi-turn video chat.<n>Second, we propose the SAMA model, which incorporates a versatile-temporal context aggregator and a Segment Model to jointly enhance fine-grained video comprehension and precise grounding capabilities.
  arXiv  Detail & Related papers  (2025-05-24T18:13:16Z)
• Breaking Down Video LLM Benchmarks: Knowledge, Spatial Perception, or True Temporal Understanding? [27.128582163847]
  We identify two major limitations that obscure whether higher scores truly indicate stronger understanding of the dynamic content in videos.<n>We propose VBenchComp, an automated pipeline that categorizes questions into different domains: LLM-Answerable, Semantic, and Temporal.
  arXiv  Detail & Related papers  (2025-05-20T13:07:55Z)
• VCRBench: Exploring Long-form Causal Reasoning Capabilities of Large Video Language Models [29.706347050700867]
  We introduce a novel benchmark named Video-based long-form Causal Reasoning (VCRBench)<n>VCRBench tests whether Large Video Language Models (LVLMs) can identify, reason about, and correctly sequence the events needed to accomplish a specific goal.<n>We propose Recognition-Reasoning Decomposition (RRD), a modular approach that breaks video-based causal reasoning into two sub-tasks of video recognition and causal reasoning.
  arXiv  Detail & Related papers  (2025-05-13T11:35:58Z)
• SF2T: Self-supervised Fragment Finetuning of Video-LLMs for Fine-Grained Understanding [23.96372422130216]
  Video-based Large Language Models (VideoVid-LLMs) have witnessed substantial advancements in recent years.<n>They struggle with fine-grained understanding, particularly in aspects such as visual dynamics and video details inquiries.<n>To tackle these shortcomings, we find that fine-tuning Video-LLMs on self-supervised fragment tasks greatly improve their fine-grained video understanding abilities.
  arXiv  Detail & Related papers  (2025-04-10T13:40:34Z)
• Narrating the Video: Boosting Text-Video Retrieval via Comprehensive Utilization of Frame-Level Captions [3.9633773442108873]
  We propose a new framework, Narrating the Video (NarVid), which strategically leverages the comprehensive information available from frame-level captions, the narration.<n>The proposed NarVid exploits narration in multiple ways: 1) feature enhancement through cross-modal interactions between narration and video, 2) query-aware adaptive filtering to suppress irrelevant or incorrect information, 3) dual-modal matching score by adding query-video similarity and query-narration similarity.
  arXiv  Detail & Related papers  (2025-03-07T07:15:06Z)
• Can LLMs Ask Good Questions? [45.54763954234726]
  We evaluate questions generated by large language models (LLMs) from context.<n>We compare them to human-authored questions across six dimensions: question type, question length, context coverage, answerability, uncommonness, and required answer length.
  arXiv  Detail & Related papers  (2025-01-07T03:21:17Z)
• VideoQA in the Era of LLMs: An Empirical Study [108.37456450182054]
  Video Large Language Models (Video-LLMs) are flourishing and has advanced many video-intuitive tasks.<n>This work conducts a timely and comprehensive study of Video-LLMs' behavior in VideoQA.<n>Our analyses demonstrate that Video-LLMs excel in VideoQA; they can correlate contextual cues and generate plausible responses to questions about varied video contents.<n>However, models falter in handling video temporality, both in reasoning about temporal content ordering and grounding QA-relevant temporal moments.
  arXiv  Detail & Related papers  (2024-08-08T05:14:07Z)
• LvBench: A Benchmark for Long-form Video Understanding with Versatile Multi-modal Question Answering [49.68215536040896]
  LvBench is a long-form video understanding benchmark for versatile multi-modal question-answering.<n>We consider videos ranging from 70 seconds to 4 hours, covering single-scene, multi-scene, and full-scene contexts.<n>Our dataset comprises 20,061 question-answer pairs sourced from 100 carefully selected movies.
  arXiv  Detail & Related papers  (2023-12-08T03:33:38Z)
• Dense Video Captioning: A Survey of Techniques, Datasets and Evaluation Protocols [53.706461356853445]
  Untrimmed videos have interrelated events, dependencies, context, overlapping events, object-object interactions, domain specificity, and other semantics worth describing. Video Captioning (DVC) aims at detecting and describing different events in a given video.
  arXiv  Detail & Related papers  (2023-11-05T01:45:31Z)
• Causalainer: Causal Explainer for Automatic Video Summarization [77.36225634727221]
  In many application scenarios, improper video summarization can have a large impact. Modeling explainability is a key concern. A Causal Explainer, dubbed Causalainer, is proposed to address this issue.
  arXiv  Detail & Related papers  (2023-04-30T11:42:06Z)
• Self-Supervised Learning for Videos: A Survey [70.37277191524755]
  Self-supervised learning has shown promise in both image and video domains. In this survey, we provide a review of existing approaches on self-supervised learning focusing on the video domain.
  arXiv  Detail & Related papers  (2022-06-18T00:26:52Z)
• iReason: Multimodal Commonsense Reasoning using Videos and Natural Language with Interpretability [0.0]
  Causality knowledge is vital to building robust AI systems. We propose iReason, a framework that infers visual-semantic commonsense knowledge using both videos and natural language captions.
  arXiv  Detail & Related papers  (2021-06-25T02:56:34Z)
• Fill-in-the-blank as a Challenging Video Understanding Evaluation Framework [19.031957183047048]
  We introduce a novel dataset consisting of 28,000 videos and fill-in-the-blank tests. We show that both a multimodal model and a strong language model have a large gap with human performance.
  arXiv  Detail & Related papers  (2021-04-09T04:00:10Z)
• Dense-Caption Matching and Frame-Selection Gating for Temporal Localization in VideoQA [96.10612095576333]
  We propose a video question answering model which effectively integrates multi-modal input sources and finds the temporally relevant information to answer questions. Our model is also comprised of dual-level attention (word/object and frame level), multi-head self-cross-integration for different sources (video and dense captions), and which pass more relevant information to gates. We evaluate our model on the challenging TVQA dataset, where each of our model components provides significant gains, and our overall model outperforms the state-of-the-art by a large margin.
  arXiv  Detail & Related papers  (2020-05-13T16:35:27Z)
• VIOLIN: A Large-Scale Dataset for Video-and-Language Inference [103.7457132841367]
  We introduce a new task, Video-and-Language Inference, for joint multimodal understanding of video and text. Given a video clip with subtitles aligned as premise, paired with a natural language hypothesis based on the video content, a model needs to infer whether the hypothesis is entailed or contradicted by the given video clip. A new large-scale dataset, named Violin (VIdeO-and-Language INference), is introduced for this task, which consists of 95,322 video-hypothesis pairs from 15,887 video clips.
  arXiv  Detail & Related papers  (2020-03-25T20:39:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.