How Far Are Video Models from True Multimodal Reasoning?

• URL: http://arxiv.org/abs/2604.19193v1
• Date: Tue, 21 Apr 2026 08:04:02 GMT
• Title: How Far Are Video Models from True Multimodal Reasoning?
• Abstract summary: CLVG-Bench is an evaluation framework designed to probe video models' zero-shot reasoning capabilities.<n>It comprises more than 1,000 high-quality, manually annotated metadata across 6 categories and 47 subcategories.<n>We propose an Adaptive Video Evaluator (AVE) that aligns with human expert perception using minimal annotations.
• Score: 32.10821745418955
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Despite remarkable progress toward general-purpose video models, a critical question remains unanswered: how far are these models from achieving true multimodal reasoning? Existing benchmarks fail to address this question rigorously, as they remain constrained by straightforward task designs and fragmented evaluation metrics that neglect complex multimodal reasoning. To bridge this gap, we introduce CLVG-Bench, an evaluation framework designed to probe video models' zero-shot reasoning capabilities via Context Learning in Video Generation. CLVG-Bench comprises more than 1,000 high-quality, manually annotated metadata across 6 categories and 47 subcategories, covering complex scenarios including physical simulation, logical reasoning, and interactive contexts. To enable rigorous and scalable assessment, we further propose an Adaptive Video Evaluator (AVE) that aligns with human expert perception using minimal annotations, delivering interpretable textual feedback across diverse video context tasks. Extensive experiments reveal a striking answer to our central question: while state-of-the-art (SOTA) video models, such as Seedance 2.0, demonstrate competence on certain understanding and reasoning subtasks, they fall substantially short with logically grounded and interactive generation tasks (achieving success rates <25% and ~0%, respectively), exposing multimodal reasoning and physical grounding as critical bottlenecks. By systematically quantifying these limitations, the proposed method provides actionable feedbacks and a clear roadmap toward truly robust, general-purpose video models. CLVG-Bench and code are released here.

Related papers

• Video-IFBench: Evaluating Instruction Following of Multimodal LLMs in Video Understanding Scenarios [61.926801508808616]
  We introduce Video-IFBench, a benchmark for evaluating instruction following in video understanding.<n>We develop an instruction taxonomy with four templates, including single-task, multi-task, selection, and nested instructions.<n>We build a semi-automatic data construction pipeline that combines MLLMs, programmatic processing, and human verification, resulting in 1.5K samples.
  arXiv  Detail & Related papers  (2026-08-26T08:35:42Z)
• Video-MME-v2: Towards the Next Stage in Benchmarks for Comprehensive Video Understanding [98.3098451637867]
  Video-MME-v2 is a comprehensive benchmark designed to rigorously evaluate the robustness and faithfulness of video understanding.<n>To guarantee data quality, Video-MME-v2 is constructed through a rigorously controlled human annotation pipeline.
  arXiv  Detail & Related papers  (2026-04-06T17:59:56Z)
• A Skill-augmented Agentic Framework and Benchmark for Multi-Video Understanding [69.31609753061137]
  Multimodal Large Language Models have achieved strong performance in single-video understanding, yet their ability to reason across multiple videos remains limited.<n>Existing approaches typicallyulate multiple videos into a single input and performulate direct inference, which introduces training-inference mismatch.<n>Current multi-video benchmarks primarily emphasize event-level comparison, leaving identity-level matching, fine-grained discrimination, and structured multi-step reasoning underexplored.<n>We propose SAMA, a Skill-Augmented Agentic Framework for Multi-Video Understanding, which integrates visual tools, task-specific skills, and a conflict-aware verification mechanism
  arXiv  Detail & Related papers  (2026-03-16T02:09:48Z)
• CVBench: Evaluating Cross-Video Synergies for Complex Multimodal Understanding and Reasoning [11.478276629279526]
  We present CVBench, the first comprehensive benchmark designed to assess cross-video relational reasoning rigorously.<n>CVBench comprises 1,000 question-answer pairs spanning three tiers: cross-video object association, cross-video event association, and cross-video complex reasoning.<n>Built from five domain-diverse video clusters, the benchmark challenges models to synthesise information across dynamic visual contexts.
  arXiv  Detail & Related papers  (2025-08-27T03:29:35Z)
• ImplicitQA: Going beyond frames towards Implicit Video Reasoning [39.63171940350552]
  ImplicitQA is a novel benchmark designed to test VideoQA models on human-like implicit reasoning.<n>ImplicitQA comprises 1K meticulously annotated QA pairs drawn from 1K high-quality creative video clips.
  arXiv  Detail & Related papers  (2025-06-26T19:53:54Z)
• VidBridge-R1: Bridging QA and Captioning for RL-based Video Understanding Models with Intermediate Proxy Tasks [41.90092896728809]
  We present VidBridge-R1, the first versatile video reasoning model that effectively bridges the "Reason-Then-Respond" paradigm conflict.<n>Extensive experiments show that VidBridge-R1 achieves significant performance gains on both QA and captioning within one model.
  arXiv  Detail & Related papers  (2025-06-10T03:57:53Z)
• VideoMathQA: Benchmarking Mathematical Reasoning via Multimodal Understanding in Videos [89.39873803375498]
  VideoMathQA is a benchmark designed to evaluate whether models can perform temporally extended cross-modal reasoning on videos.<n>The benchmark spans 10 diverse mathematical domains, covering videos ranging from 10 seconds to over 1 hour.<n>It requires models to interpret structured visual content, understand instructional narratives, and jointly ground concepts across visual, audio, and textual modalities.
  arXiv  Detail & Related papers  (2025-06-05T17:59:58Z)
• MMR-V: What's Left Unsaid? A Benchmark for Multimodal Deep Reasoning in Videos [22.10711693948861]
  We propose MMR-V: A Benchmark for Multimodal Deep Reasoning in Videos.<n>The benchmark is characterized by the following features.<n>Experiments reveal that current models still struggle with multi-modal reasoning.
  arXiv  Detail & Related papers  (2025-06-04T16:33:41Z)
• SAMA: Towards Multi-Turn Referential Grounded Video Chat with Large Language Models [93.73583158211115]
  Achieving fine-grained-temporal understanding in videos remains a major challenge for current Video Large Multimodels (Video LMMs)<n>We contribute in three core aspects: dataset, model, and benchmark.<n>First, we introduce SAMA-239K, a large-scale dataset comprising 15K videos specifically to enable joint learning of video understanding, grounding, and multi-turn video chat.<n>Second, we propose the SAMA model, which incorporates a versatile-temporal context aggregator and a Segment Model to jointly enhance fine-grained video comprehension and precise grounding capabilities.
  arXiv  Detail & Related papers  (2025-05-24T18:13:16Z)
• STEP: Enhancing Video-LLMs' Compositional Reasoning by Spatio-Temporal Graph-guided Self-Training [87.58996020705258]
  Video Large Language Models (Video-LLMs) have recently shown strong derivation in basic video understanding tasks.<n>Video-LLMs struggle with compositional reasoning that requires multi-step explicit-temporal inference across object relations, interactions and events.<n>We propose STEP, a novel graph-guided self-training method that enables VideoLLMs to generate reasoning-rich finetuning data from any raw videos to improve itself.
  arXiv  Detail & Related papers  (2024-11-29T11:54:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.