NEST: Narrative Event Structures in Time for Long Video Understanding

• URL: http://arxiv.org/abs/2606.19706v1
• Date: Thu, 18 Jun 2026 02:05:14 GMT
• Title: NEST: Narrative Event Structures in Time for Long Video Understanding
• Abstract summary: We introduce NEST (Narrative Event Structures in Time for Long Video Understanding), a dataset of 1005 full-length movies (avg. 98 minutes)<n>NEST captures multimodal narrative events with structured annotations grounded in visual content, dialogue, and audio, and links them through relations that reflect narrative structure.<n>The benchmark is highly challenging for grounded event discovery, with ETD below 8%, EL under 6%, and EAE below 11%.
• Score: 6.938433277109506
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent progress in vision-language models has enabled the processing of increasingly long video sequences, but the ability to handle extended token streams does not translate to understanding of narrative structure in long videos. Existing long video benchmarks focus on needle-in-a-haystack retrieval rather than evaluating how low-level actions form events, how events interact across time, and how narratives progress, for example, whether a model can connect an early setback, such as a job loss to a later relationship breakup, despite long gaps, intervening scenes, or flashbacks that reframe what occurred. We introduce NEST (Narrative Event Structures in Time for Long Video Understanding), a dataset of 1005 full-length movies (avg. 98 minutes), each annotated with 102 multimodal narrative events grounded in visual content, dialogue, and audio. NEST captures multimodal narrative events with structured annotations grounded in visual content, dialogue, and audio, and links them through relations that reflect narrative structure, including temporal ordering, hierarchical composition, and long-range dependencies. We introduce baselines for event trigger detection (ETD), event localization (EL), event argument extraction (EAE), and event relation extraction (ERE). The benchmark is highly challenging for grounded event discovery, with ETD below 8%, EL under 6%, and EAE below 11%. In contrast, ERE is more tractable once events are given, reaching 35.45% F1 zero-shot and 44.42% F1 after fine-tuning.

Related papers

• MELON: A Large-Scale Dataset for Multi-Event Text-to-Long-Video Retrieval [9.116098272868667]
  Existing text-video retrieval datasets primarily consist of short-form clips containing a single dominant event.<n>We introduce MELON, the first large-scale dataset designed to extend text-video retrieval to long-form videos featuring complex, multi-event structures.
  arXiv  Detail & Related papers  (2026-08-31T08:20:35Z)
• Beyond Visual Boundaries: Rethinking Scene Segmentation for Movie RAG [26.7662002828488]
  Retrieval-augmented generation (RAG) offers a promising middle ground by selectively retrieving relevant video segments for grounded generation.<n>We investigate RAG for movie understanding, which demands story-level reasoning over characters, events, and narrative arcs spanning hours of content.<n>We introduce NarraScene, a narrative-centric scene segmentation dataset annotated with a three-level cognitive taxonomy spanning physical, character, and narrative change.
  arXiv  Detail & Related papers  (2026-08-27T14:25:32Z)
• Event ActivityNet: A Large-Scale Simulated-Event Benchmark for Untrimmed Action Understanding [26.582838811672413]
  Event ActivityNet is a large-scale simulated-event benchmark from human-annotated, untrimmed ActivityNet videos.<n>It supports annotated-segment action recognition, auxiliary event-language alignment, and causal online temporal action localization.<n>Under a progressive nested-scale training protocol, recognition Top-1 accuracy increases from 52.25 to 66.42, while online temporal localization average mAP improves from 21.7 to 29.0.
  arXiv  Detail & Related papers  (2026-08-03T09:20:46Z)
• Narrative Aligned Long Form Video Question Answering [12.805359127629215]
  We introduce NA-VQA, a benchmark designed to evaluate deep temporal and narrative reasoning in long-form videos.<n> NA-VQA contains 88 full-length movies and 4.4K open-ended question-answer pairs, each grounded in multiple evidence spans labeled as Short, Medium, or Far.<n>We propose Video-NaRA, a narrative-centric framework that builds event-level chains and stores them in a structured memory for retrieval during reasoning.
  arXiv  Detail & Related papers  (2026-03-19T21:23:15Z)
• TimeChat-Captioner: Scripting Multi-Scene Videos with Time-Aware and Structural Audio-Visual Captions [64.27159505605312]
  This paper proposes Omni Captioning, a novel task designed to generate continuous, fine-grained, and structured audio-visual narratives with explicit timestamps.<n>To ensure dense semantic coverage, we introduce a six-dimensional structural schema to create "script-like" captions.<n>Extensive experiments demonstrate that TimeChat-Captioner-7B achieves state-of-the-art performance, surpassing Gemini-2.5-Pro.
  arXiv  Detail & Related papers  (2026-02-09T14:21:58Z)
• LET-US: Long Event-Text Understanding of Scenes [23.376693904132786]
  Event cameras output event streams as sparse, asynchronous data with microsecond-level temporal resolution.<n>We introduce LET-US, a framework for long event-stream--text comprehension.<n>We use an adaptive compression mechanism to reduce the volume of input events while preserving critical visual details.
  arXiv  Detail & Related papers  (2025-08-10T16:02:41Z)
• SceneRAG: Scene-level Retrieval-Augmented Generation for Video Understanding [6.980340270823506]
  We present SceneRAG, a framework to segment videos into narrative-consistent scenes.<n>For each scene, the framework fuses information from both visual and textual modalities to extract entity relations.<n>Experiments on the LongerVideos benchmark, featuring over 134 hours of diverse content, confirm that SceneRAG substantially outperforms prior baselines.
  arXiv  Detail & Related papers  (2025-06-09T10:00:54Z)
• TEMPURA: Temporal Event Masked Prediction and Understanding for Reasoning in Action [28.930109403769166]
  We propose TEMPURA, a two-stage training framework that enhances video temporal understanding.<n>TEMPURA first applies masked event prediction reasoning to reconstruct missing events and generate step-by-step causal explanations from dense event annotations.<n>We train TEMPURA on VER, a large-scale dataset curated by us that comprises 1M training instances and 500K videos with temporally aligned event descriptions and structured reasoning steps.
  arXiv  Detail & Related papers  (2025-05-02T21:00:17Z)
• Everything Can Be Described in Words: A Simple Unified Multi-Modal Framework with Semantic and Temporal Alignment [0.0]
  We propose UMaT, a framework that unifies visual and auditory inputs as structured text for large language models.<n>It significantly improves state-of-the-art Long Video Question Answering accuracy.
  arXiv  Detail & Related papers  (2025-03-12T05:28:24Z)
• Grounding Partially-Defined Events in Multimodal Data [61.0063273919745]
  We introduce a multimodal formulation for partially-defined events and cast the extraction of these events as a three-stage span retrieval task. We propose a benchmark for this task, MultiVENT-G, that consists of 14.5 hours of densely annotated current event videos and 1,168 text documents, containing 22.8K labeled event-centric entities. Results illustrate the challenges that abstract event understanding poses and demonstrates promise in event-centric video-language systems.
  arXiv  Detail & Related papers  (2024-10-07T17:59:48Z)
• EA-VTR: Event-Aware Video-Text Retrieval [97.30850809266725]
  Event-Aware Video-Text Retrieval model achieves powerful video-text retrieval ability through superior video event awareness. EA-VTR can efficiently encode frame-level and video-level visual representations simultaneously, enabling detailed event content and complex event temporal cross-modal alignment.
  arXiv  Detail & Related papers  (2024-07-10T09:09:58Z)
• Analyzing Temporal Complex Events with Large Language Models? A Benchmark towards Temporal, Long Context Understanding [57.62275091656578]
  We refer to the complex events composed of many news articles over an extended period as Temporal Complex Event (TCE) This paper proposes a novel approach using Large Language Models (LLMs) to systematically extract and analyze the event chain within TCE.
  arXiv  Detail & Related papers  (2024-06-04T16:42:17Z)
• HierVL: Learning Hierarchical Video-Language Embeddings [108.77600799637172]
  HierVL is a novel hierarchical video-language embedding that simultaneously accounts for both long-term and short-term associations. We introduce a hierarchical contrastive training objective that encourages text-visual alignment at both the clip level and video level. Our hierarchical scheme yields a clip representation that outperforms its single-level counterpart as well as a long-term video representation that achieves SotA.
  arXiv  Detail & Related papers  (2023-01-05T21:53:19Z)
• Spoken Moments: Learning Joint Audio-Visual Representations from Video Descriptions [75.77044856100349]
  We present the Spoken Moments dataset of 500k spoken captions each attributed to a unique short video depicting a broad range of different events. We show that our AMM approach consistently improves our results and that models trained on our Spoken Moments dataset generalize better than those trained on other video-caption datasets.
  arXiv  Detail & Related papers  (2021-05-10T16:30:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.