StoryTR: Narrative-Centric Video Temporal Retrieval with Theory of Mind Reasoning

• URL: http://arxiv.org/abs/2604.23198v1
• Date: Sat, 25 Apr 2026 08:09:31 GMT
• Title: StoryTR: Narrative-Centric Video Temporal Retrieval with Theory of Mind Reasoning
• Abstract summary: Current video moment retrieval excels at action-centric tasks but struggles with narrative content.<n>We introduce textbfStoryTR, the first video moment retrieval benchmark requiring ToM reasoning.<n>Our 7B textbfShorts-Moment model, trained on ToM-guided data, improves +15.1% relative IoU over baselines.
• Score: 9.122764358915276
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Current video moment retrieval excels at action-centric tasks but struggles with narrative content. Models can see \textit{what is happening} but fail to reason \textit{why it matters}. This semantic gap stems from the lack of \textbf{Theory of Mind (ToM)}: the cognitive ability to infer implicit intentions, mental states, and narrative causality from surface-level observations. We introduce \textbf{StoryTR}, the first video moment retrieval benchmark requiring ToM reasoning, comprising 8.1k samples from narrative short-form videos (shorts/reels). These videos present an ideal testbed. Their high information density encodes meaning through subtle multimodal cues. For instance, a glance paired with a sigh carries entirely different semantics than the glance alone. Yet multimodal perception alone is insufficient; ToM is required to decode that a character ``smiling'' may actually be ``concealing hostility.'' To teach models this reasoning capability, we propose an \textbf{Agentic Data Pipeline} that generates training data with explicit three-tier ToM chains (intent decoding, narrative reasoning, boundary localization). Experiments reveal the severity of the reasoning gap: Gemini-3.0-Pro achieves only 0.53 Avg IoU on StoryTR. However, our 7B \textbf{Shorts-Moment} model, trained on ToM-guided data, improves +15.1\% relative IoU over baselines, demonstrating that \textit{narrative reasoning capability matters more than parameter scale}.

Related papers

• Video-HolmesV2: Can MLLMs Reason with Spatio-Temporal Audio-Visual Evidence in Long Videos? [22.544360578469156]
  We present Video-Holmes-V2, a novel benchmark for Deep Audio-Visual Coupling.<n>It enforces an Evidence-Based Evaluation, requiring models to justify answers with precise-temporal audio-visual evidence.<n>Our method distills high-value reasoning cues to mitigate long-grained noise.
  arXiv  Detail & Related papers  (2026-09-15T14:30:00Z)
• Perception First: A Frontier Native-Video Model with Self-Consistency for Implicit Video Question Answering [0.0]
  We describe our submission to the VRR Challenge @ CVPR 2026, built on the emphImplicitQA / emphVRR-QA benchmarkciteimplicitqa.<n>Our central finding is that this benchmark is emphperception-bound rather than reasoning-bound.
  arXiv  Detail & Related papers  (2026-05-31T23:00:17Z)
• SinkTrack: Attention Sink based Context Anchoring for Large Language Models [57.87121346197518]
  Large language models (LLMs) suffer from hallucination and context forgetting.<n>SinkTrack treats BOS> as an information anchor and injects key contextual features into its representation.<n>SinkTrack is training-free, plug-and-play, and introduces negligible inference overhead.
  arXiv  Detail & Related papers  (2026-04-11T04:49:11Z)
• MME-CoF-Pro: Evaluating Reasoning Coherence in Video Generative Models with Text and Visual Hints [95.27042253462963]
  MME-CoF-Pro is a benchmark to assess reasoning coherence in video models.<n>MME-CoF-Pro contains 303 samples across 16 categories, ranging from visual logical to scientific reasoning.
  arXiv  Detail & Related papers  (2026-03-20T17:59:56Z)
• Narrative Aligned Long Form Video Question Answering [12.805359127629215]
  We introduce NA-VQA, a benchmark designed to evaluate deep temporal and narrative reasoning in long-form videos.<n> NA-VQA contains 88 full-length movies and 4.4K open-ended question-answer pairs, each grounded in multiple evidence spans labeled as Short, Medium, or Far.<n>We propose Video-NaRA, a narrative-centric framework that builds event-level chains and stores them in a structured memory for retrieval during reasoning.
  arXiv  Detail & Related papers  (2026-03-19T21:23:15Z)
• Scale Can't Overcome Pragmatics: The Impact of Reporting Bias on Vision-Language Reasoning [79.95774256444956]
  The lack of reasoning capabilities in Vision-Language Models has remained at the forefront of research discourse.<n>We investigate the data underlying the popular VLMs OpenCLIP, LLaVA-1.5 and Molmo through the lens of theories from pragmatics.
  arXiv  Detail & Related papers  (2026-02-26T18:54:06Z)
• ClueTracer: Question-to-Vision Clue Tracing for Training-Free Hallucination Suppression in Multimodal Reasoning [11.30122927323085]
  We introduce ClueRecall, a metric for assessing visual clue retrieval.<n>We present ClueTracer, a training-free, parameter-free, and architecture-agnostic plugin for hallucination suppression.
  arXiv  Detail & Related papers  (2026-02-02T12:03:56Z)
• NOAH: Benchmarking Narrative Prior driven Hallucination and Omission in Video Large Language Models [8.6767620170781]
  Video large language models (Video LLMs) have recently achieved strong performance on tasks such as captioning, summarization, and question answering.<n>Many models and training methods explicitly encourage continuity across events to enhance narrative coherence.<n>We identify this bias, which we call narrative prior, as a key driver of two errors: hallucinations, where non-existent events are introduced or existing ones are misinterpreted, and omissions, where factual events are suppressed because they are misaligned with surrounding context.
  arXiv  Detail & Related papers  (2025-11-09T17:41:11Z)
• BLINK-Twice: You see, but do you observe? A Reasoning Benchmark on Visual Perception [67.89135437537179]
  We introduce BLINK-Twice, a vision-centric reasoning benchmark grounded in challenging perceptual tasks.<n>Instead of relying on external knowledge, our tasks require models to reason from visual content alone.<n>Compared to prior perception benchmarks, it moves beyond shallow perception and requires fine-grained observation and analytical reasoning.
  arXiv  Detail & Related papers  (2025-10-10T13:14:13Z)
• Thinking Before You Speak: A Proactive Test-time Scaling Approach [54.8205006555199]
  We implement our idea as a reasoning framework, named emphThinking Before You Speak (TBYS)<n>We design a pipeline for automatically collecting and filtering in-context examples for the generation of emphinsights.<n>Experiments on challenging mathematical datasets verify the effectiveness of TBYS.
  arXiv  Detail & Related papers  (2025-08-26T03:43:32Z)
• Movie Facts and Fibs (MF$^2$): A Benchmark for Long Movie Understanding [97.05584099530226]
  We introduce MF$2$, a new benchmark for evaluating whether models can comprehend, consolidate, and recall key narrative information from full-length movies.<n>For each pair, models must correctly identify both the true and false claims.<n>Our experiments demonstrate that both open-weight and closed state-of-the-art models fall well short of human performance.
  arXiv  Detail & Related papers  (2025-06-06T17:58:36Z)
• StoryGPT-V: Large Language Models as Consistent Story Visualizers [33.68157535461168]
  generative models have demonstrated impressive capabilities in generating realistic and visually pleasing images grounded on textual prompts.<n>Yet, the emerging Large Language Model (LLM) showcases robust reasoning abilities to navigate through ambiguous references.<n>We introduce emphStoryGPT-V, which leverages the merits of the latent diffusion (LDM) and LLM to produce images with consistent and high-quality characters.
  arXiv  Detail & Related papers  (2023-12-04T18:14:29Z)
• Dual-Level Decoupled Transformer for Video Captioning [15.193977761440404]
  Video captioning aims to understand the video semantic concept of the video and generate descriptive sentences. mathcalD2$ is a dual-level vision decoupled transformer pipeline. emph(i) for video-temporal representation, we decouple the process of it into "first-spatial-then-temporal" paradigm.
  arXiv  Detail & Related papers  (2022-05-06T06:37:07Z)
• Learning to Discretely Compose Reasoning Module Networks for Video Captioning [81.81394228898591]
  We propose a novel visual reasoning approach for video captioning, named Reasoning Module Networks (RMN) RMN employs three sophisticated RM-temporal reasoning, and 2) a dynamic and discrete module selector trained by a linguistic loss with a Gumbel approximation.
  arXiv  Detail & Related papers  (2020-07-17T15:27:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.