Video-MME-Logical: A Controlled Diagnostic Benchmark for Video Temporal-Logical Reasoning

• URL: http://arxiv.org/abs/2606.27828v1
• Date: Fri, 26 Jun 2026 08:12:04 GMT
• Title: Video-MME-Logical: A Controlled Diagnostic Benchmark for Video Temporal-Logical Reasoning
• Abstract summary: Video temporal-logical reasoning requires models to maintain, update, and compose evidence as visual states evolve across frames.<n>We introduce Video-MME-Logical, a benchmark organized around five temporal-logical operations.<n>It enables difficulty-controlled final-answer evaluation by varying temporal horizon and reasoning complexity.
• Score: 28.325120357282156
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent interest in multimodal large language models (MLLMs) raises a central question: can they reason over dynamic visual evidence rather than merely recognize objects or events in individual frames? This ability, which we refer to as video temporal-logical reasoning, requires models to maintain, update, and compose evidence as visual states evolve across frames. Existing video benchmarks often conflate this capability with scene complexity, static recognition, or uncontrolled temporal variation. To isolate this capability, we introduce Video-MME-Logical, a controlled benchmark organized around five temporal-logical operations: state tracking, sequential counting, temporal ordering, dynamic spatiality, and structural composition. The benchmark contains 25 fine-grained task categories generated with controlled object states, transitions, temporal dependencies, and logical compositions. It enables difficulty-controlled final-answer evaluation by varying temporal horizon and reasoning complexity, and supports intermediate-state diagnostics by verifying whether models recover the required logical reasoning trace before producing the final answer. Experiments with state-of-the-art MLLMs reveal a substantial human-model gap, especially as temporal-logical complexity increases. Supervised fine-tuning on up to 500K generated samples improves performance but remains insufficient to close the reasoning gap, positioning Video-MME-Logical as a scalable testbed for analyzing and improving temporal-logical reasoning in MLLMs.

Related papers

• Moment-Video: Diagnosing Temporal Fidelity of Video MLLMs on Momentary Visual Events [52.031070006859544]
  Video multimodal large language models (MLLMs) have made rapid progress on general and long-form video understanding, yet their ability to preserve brief answer-critical visual evidence remains underexplored.<n>We introduce Moment-Video, a benchmark for diagnosing the temporal fidelity of video MLLMs through momentary visual event understanding.
  arXiv  Detail & Related papers  (2026-06-01T17:32:20Z)
• STORM: Internalized Modeling for Spatial-Temporal Reasoning in Video-Language Models [81.32710031596591]
  Video reasoning tasks require tracking motion, temporal order, and evolving visual states across frames.<n>Existing methods built on vision-language models (LVLMs) often address this challenge by externalizing reasoning through chain-of-thought (CoT)<n>We propose STORMS, a two-stage framework that teaches LVLMs to reason through bounded continuous latent trajectories instead of explicit textual CoT.
  arXiv  Detail & Related papers  (2026-05-25T16:33:00Z)
• TOC-Bench: A Temporal Object Consistency Benchmark for Video Large Language Models [9.648992690108086]
  Video large language models (Video-LLMs) have made strong progress in general video understanding, but their ability to maintain temporal object consistency remains underexplored.<n>We introduce TOC-Bench, a diagnostic benchmark for evaluating temporal object consistency in Video-LLMs.
  arXiv  Detail & Related papers  (2026-05-11T02:47:59Z)
• Tracking the Truth: Object-Centric Spatio-Temporal Monitoring for Video Large Language Models [154.39583176906893]
  Large language models (MLs) have advanced understanding, they remain highly prone to hallucinations in scenes.<n>We argue this stems from a failure intemporal monitoring, the ability to persistently track object dynamic identities, states, and relations over time.<n>Existing benchmarks obscure this deficit relying on single final-answer queries that can often be resolved via local visual cues or statistical priors.
  arXiv  Detail & Related papers  (2026-05-09T14:32:36Z)
• TimeBlind: A Spatio-Temporal Compositionality Benchmark for Video LLMs [24.299498301173255]
  TimeBlind is a diagnostic benchmark for fine-grained temporal understanding.<n>We evaluate over 20 state-of-the-art MLLMs on 600 instances.<n>The Instance Accuracy of the best performing MLLM is only 48.2%, far below the human performance (98.2%)
  arXiv  Detail & Related papers  (2026-01-30T20:21:46Z)
• MORSE-500: A Programmatically Controllable Video Benchmark to Stress-Test Multimodal Reasoning [54.47710436807661]
  MORSE-500 is a video benchmark composed of 500 fully scripted clips embedded questions spanning six complementary reasoning categories.<n>Each instance is generated using deterministic Python scripts (Manim, Matplotlib, MoviePy), generative video models, and real footage.<n>Unlike static benchmarks that become obsolete once saturated, MORSE-500 is built to evolve.
  arXiv  Detail & Related papers  (2025-06-05T19:12:45Z)
• V-STaR: Benchmarking Video-LLMs on Video Spatio-Temporal Reasoning [40.18308199837137]
  We introduce a Video S-Temporal Reasoning (V-STa) benchmark to address these shortcomings.<n>We construct a dataset to elicit the spatial-temporal reasoning process of Video-LLMs.<n>Experiments from 14 Video-LLMs reveal significant gaps between current Video-LLMs and the needs for robust and consistent consistent reasoning.
  arXiv  Detail & Related papers  (2025-03-14T15:21:44Z)
• TOMATO: Assessing Visual Temporal Reasoning Capabilities in Multimodal Foundation Models [55.48403691519395]
  TOMATO is a novel benchmark crafted to rigorously assess MFMs' temporal reasoning capabilities in video understanding.<n>TOMATO comprises 1,484 carefully curated, human-annotated questions spanning six tasks.<n>Our comprehensive evaluation reveals a human-model performance gap of 57.3% with the best-performing model.
  arXiv  Detail & Related papers  (2024-10-30T17:50:23Z)
• TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Models [75.42002690128486]
  TemporalBench is a new benchmark dedicated to evaluating fine-grained temporal understanding in videos. It consists of 10K video question-answer pairs, derived from 2K high-quality human annotations detailing the temporal dynamics in video clips. Results show that state-of-the-art models like GPT-4o achieve only 38.5% question answering accuracy on TemporalBench.
  arXiv  Detail & Related papers  (2024-10-14T17:59:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.