Video-MME-v2: Towards the Next Stage in Benchmarks for Comprehensive Video Understanding

• URL: http://arxiv.org/abs/2604.05015v1
• Date: Mon, 06 Apr 2026 17:59:56 GMT
• Title: Video-MME-v2: Towards the Next Stage in Benchmarks for Comprehensive Video Understanding
• Abstract summary: Video-MME-v2 is a comprehensive benchmark designed to rigorously evaluate the robustness and faithfulness of video understanding.<n>To guarantee data quality, Video-MME-v2 is constructed through a rigorously controlled human annotation pipeline.
• Score: 98.3098451637867
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: With the rapid advancement of video understanding, existing benchmarks are becoming increasingly saturated, exposing a critical discrepancy between inflated leaderboard scores and real-world model capabilities. To address this widening gap, we introduce Video-MME-v2, a comprehensive benchmark designed to rigorously evaluate the robustness and faithfulness of video understanding. To systematically evaluate model capabilities, we design a \textbf{progressive tri-level hierarchy} that incrementally increases the complexity of video comprehension, ranging from multi-point visual information aggregation, to temporal dynamics modeling, and ultimately to complex multimodal reasoning. Besides, in contrast to conventional per-question accuracy, we propose a \textbf{group-based non-linear evaluation} strategy that enforces both consistency across related queries and coherence in multi-step reasoning. It penalizes fragmented or guess-based correctness and assigns credit only to answers supported by valid reasoning. To guarantee data quality, Video-MME-v2 is constructed through a rigorously controlled human annotation pipeline, involving 12 annotators and 50 independent reviewers. Backed by \textbf{3,300 human-hours} and up to \textbf{5 rounds} of quality assurance, Video-MME-v2 aims to serve as one of the most authoritative video benchmarks. Extensive experiments reveal a substantial gap between current best model Gemini-3-Pro and human experts, and uncover a clear hierarchical bottleneck where errors in visual information aggregation and temporal modeling propagate to limit high-level reasoning. We further find that thinking-based reasoning is highly dependent on textual cues, improving performance with subtitles but sometimes degrading it in purely visual settings. By exposing these limitations, Video-MME-v2 establishes a demanding new testbed for the development of next-generation video MLLMs.

Related papers

• Kwai Keye-VL-2.0 Technical Report [53.82434681649277]
  Keye-VL-2.0 is a multimodal foundation model designed to advance long-video understanding and agentic intelligence.<n>It is the first to adapt DeepSeek Sparse Attention (DSA) to GQA-based multimodal architectures.<n>Cross-Modal Multi-Teacher On-Policy Distillation (MOPD) paired with Context-RL and Video-RL overcomes the algorithmic dilemma of catastrophic forgetting.
  arXiv  Detail & Related papers  (2026-06-09T09:58:08Z)
• HAVEN: Hierarchically Aligned Multimodal Benchmark for Unified Video Understanding [1.8482679687103294]
  HAVEN is a hierarchically aligned multimodal benchmark for unified video understanding.<n>Built upon this unified annotation paradigm, we propose a comprehensive evaluation suite spanning summarization, temporal reasoning, multimodal grounding, and saliency ranking.
  arXiv  Detail & Related papers  (2026-05-19T00:48:14Z)
• MSVBench: Towards Human-Level Evaluation of Multi-Shot Video Generation [48.84450712826316]
  MSVBench is the first comprehensive benchmark featuring hierarchical scripts and reference images tailored for Multi-Shot Video generation.<n>We propose a hybrid evaluation framework that synergizes the high-level semantic reasoning of Large Multimodal Models with the fine-grained perceptual rigor of domain-specific expert models.
  arXiv  Detail & Related papers  (2026-02-27T12:26:34Z)
• UI2V-Bench: An Understanding-based Image-to-video Generation Benchmark [35.157850129371525]
  Image-to-Video (I2V) generation has become a major focus in the field of video synthesis.<n>Existing evaluation benchmarks primarily focus on aspects such as video quality and temporal consistency.<n>We propose UI2V-Bench, a novel benchmark for evaluating I2V models with a focus on semantic understanding and reasoning.
  arXiv  Detail & Related papers  (2025-09-29T08:14:26Z)
• VideoScore2: Think before You Score in Generative Video Evaluation [69.43069741467603]
  VideoScore2 is a multi-dimensional, interpretable, and human-aligned framework that explicitly evaluates visual quality, text-to-video alignment, and physical/common-sense consistency.<n>Our model is trained on a large-scale dataset VideoFeedback2 containing 27,168 human-annotated videos.
  arXiv  Detail & Related papers  (2025-09-26T18:09:03Z)
• HumanVideo-MME: Benchmarking MLLMs for Human-Centric Video Understanding [120.84817886550765]
  Multimodal Large Language Models (MLLMs) have demonstrated significant advances in visual understanding tasks involving both images and videos.<n>Existing human-centric benchmarks predominantly emphasize video generation quality and action recognition, while overlooking essential perceptual and cognitive abilities required in human-centered scenarios.<n>We propose a rigorously curated benchmark designed to provide a more holistic evaluation of MLLMs in human-centric video understanding.
  arXiv  Detail & Related papers  (2025-07-07T11:52:24Z)
• Bridging Video Quality Scoring and Justification via Large Multimodal Models [14.166920184033463]
  Classical video quality assessment (VQA) methods generate a numerical score to judge a video's perceived visual fidelity and clarity.<n>But a score fails to describe the video's complex quality dimensions, restricting its applicability.<n>Benefiting from the linguistic output, adapting video large multimodal models (LMMs) to VQA via instruction tuning has the potential to address this issue.
  arXiv  Detail & Related papers  (2025-06-26T05:02:25Z)
• VideoMolmo: Spatio-Temporal Grounding Meets Pointing [66.19964563104385]
  VideoMolmo is a model tailored for fine-grained pointing of video sequences.<n>A novel temporal mask fusion employs SAM2 for bidirectional point propagation.<n>To evaluate the generalization of VideoMolmo, we introduce VPoMolS-temporal, a challenging out-of-distribution benchmark spanning five real-world scenarios.
  arXiv  Detail & Related papers  (2025-06-05T17:59:29Z)
• H2VU-Benchmark: A Comprehensive Benchmark for Hierarchical Holistic Video Understanding [25.111988967973147]
  Existing benchmarks for evaluating video understanding exhibit significant limitations in coverage, task diversity, and scene adaptability.<n>We propose a hierarchical and holistic video understanding benchmark designed to evaluate both general video and online streaming video comprehension.<n>This benchmark contributes three key features: extended video duration, comprehensive assessment tasks, andEnriched video data.
  arXiv  Detail & Related papers  (2025-03-31T12:32:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.