Reasoning is All You Need for Video Generalization: A Counterfactual Benchmark with Sub-question Evaluation

• URL: http://arxiv.org/abs/2503.10691v2
• Date: Wed, 04 Jun 2025 05:57:18 GMT
• Title: Reasoning is All You Need for Video Generalization: A Counterfactual Benchmark with Sub-question Evaluation
• Authors: Qiji Zhou, Yifan Gong, Guangsheng Bao, Hongjie Qiu, Jinqiang Li, Xiangrong Zhu, Huajian Zhang, Yue Zhang,
• Abstract summary: We introduce textbfCOVER (textbfunderlineCOunterfactual textbfunderlineEo textbfunderlineReasoning), a multidimensional multimodal benchmark.<n>It decomposes complex queries into structured sub-questions, enabling fine-grained reasoning analysis.
• Score: 19.46864730994867
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Counterfactual reasoning is crucial for robust video understanding but remains underexplored in existing multimodal benchmarks. In this paper, we introduce \textbf{COVER} (\textbf{\underline{CO}}unterfactual \textbf{\underline{V}}id\textbf{\underline{E}}o \textbf{\underline{R}}easoning), a multidimensional multimodal benchmark that systematically evaluates MLLMs across the abstract-concrete and perception-cognition dimensions. Beyond prior multimodal benchmarks, COVER decomposes complex queries into structured sub-questions, enabling fine-grained reasoning analysis. Experiments on commercial and open-source models reveal a strong correlation between sub-question accuracy and counterfactual reasoning performance, highlighting the role of structured inference in video understanding. Furthermore, our results suggest a key insight: enhancing the reasoning capability of models is essential for improving the robustness of video understanding. COVER establishes a new standard for assessing MLLMs' logical reasoning abilities in dynamic environments. Our work is available at https://github.com/gongyifan-hash/COVER-Benchmark.

Related papers

• Team of One: Cracking Complex Video QA with Model Synergy [24.75732964829523]
  We propose a novel framework for open-ended video question answering that enhances reasoning depth and robustness in complex real-world scenarios.<n>Existing Video-Large Multimodal Models (Video-LMMs) often exhibit limited contextual understanding, weak temporal modeling, and poor generalization to ambiguous or compositional queries.
  arXiv  Detail & Related papers  (2025-07-18T11:12:44Z)
• Can Video Large Multimodal Models Think Like Doubters-or Double-Down: A Study on Defeasible Video Entailment [19.682019558287973]
  We introduce Defeasible Video Entailment (DVidE), a new task that challenges models to think like doubters.<n>In DVidE, given a video premise and a textual hypothesis, models must determine whether a new update strengthens or weakens the hypothesis.<n>For the generation task, we develop a framework that combines ASR output with a Large Language Model (LLM) to produce coherent, contextually relevant updates.
  arXiv  Detail & Related papers  (2025-06-27T16:51:15Z)
• PixelThink: Towards Efficient Chain-of-Pixel Reasoning [70.32510083790069]
  PixelThink is a simple yet effective scheme that integrates externally estimated task difficulty and internally measured model uncertainty.<n>It learns to compress reasoning length in accordance with scene complexity and predictive confidence.<n> Experimental results demonstrate that the proposed approach improves both reasoning efficiency and overall segmentation performance.
  arXiv  Detail & Related papers  (2025-05-29T17:55:49Z)
• KORGym: A Dynamic Game Platform for LLM Reasoning Evaluation [78.96590724864606]
  We introduce the Knowledge Orthogonal Reasoning Gymnasium (KORGym), a dynamic evaluation platform inspired by KOR-Bench and Gymnasium.<n>KORGym offers over fifty games in either textual or visual formats and supports interactive, multi-turn assessments with reinforcement learning scenarios.
  arXiv  Detail & Related papers  (2025-05-20T16:06:32Z)
• Computational Reasoning of Large Language Models [51.629694188014064]
  We introduce textbfTuring Machine Bench, a benchmark to assess the ability of Large Language Models (LLMs) to execute reasoning processes.<n> TMBench incorporates four key features: self-contained and knowledge-agnostic reasoning, a minimalistic multi-step structure, controllable difficulty, and a theoretical foundation based on Turing machine.
  arXiv  Detail & Related papers  (2025-04-29T13:52:47Z)
• Exploring the Effect of Reinforcement Learning on Video Understanding: Insights from SEED-Bench-R1 [53.894789613838654]
  We introduce SEED-Bench-R1, a benchmark designed to evaluate post-training methods for MLLMs in video understanding. It includes intricate real-world videos and complex everyday planning tasks in the format of multiple-choice questions. Using Qwen2-VL-Instruct-7B as a base model, we compare RL with supervised fine-tuning (SFT) Our detailed analysis reveals that RL enhances visual perception but often produces less coherent reasoning chains.
  arXiv  Detail & Related papers  (2025-03-31T17:55:23Z)
• QuoTA: Query-oriented Token Assignment via CoT Query Decouple for Long Video Comprehension [86.0749609778104]
  We propose QuoTA, an ante-hoc training-free modular that extends existing large video-language models. QuoTA strategically allocates frame-level importance scores based on query relevance. We decouple the query through Chain-of-Thoughts reasoning to facilitate more precise LVLM-based frame importance scoring.
  arXiv  Detail & Related papers  (2025-03-11T17:59:57Z)
• CryptoX : Compositional Reasoning Evaluation of Large Language Models [18.927129952741904]
  We introduce CryptoX, an evaluation framework that combines existing benchmarks and cryptographic.<n>We conduct detailed experiments on widely used open-source and closed-source LLMs using CryptoBench.<n>We highlight the value of independently studying compositional reasoning and emphasize the need to enhance the compositional reasoning capabilities of LLMs.
  arXiv  Detail & Related papers  (2025-02-08T17:19:43Z)
• A NotSo Simple Way to Beat Simple Bench [0.0]
  This paper presents a novel framework for enhancing reasoning capabilities in large language models (LLMs)<n>We propose a multi-step prompting strategy coupled with global consistency checks to improve model accuracy and robustness.<n>Our results reveal model-specific strengths: Claude excels in maintaining logical consistency, while GPT-4o exhibits exploratory creativity but struggles with ambiguous prompts.
  arXiv  Detail & Related papers  (2024-12-12T16:04:31Z)
• Enhancing Video-LLM Reasoning via Agent-of-Thoughts Distillation [32.930999188946345]
  This paper tackles the problem of video question answering (VideoQA)<n>Large video-language models perform well on benchmarks, but they often lack explainability and spatial-temporal grounding.<n>We propose Agent-of-Thoughts Distillation (AoTD), a method that enhances models by incorporating automatically generated Chain-of-Thoughts (CoTs) into the instruction-tuning process.
  arXiv  Detail & Related papers  (2024-12-02T16:37:50Z)
• STEP: Enhancing Video-LLMs' Compositional Reasoning by Spatio-Temporal Graph-guided Self-Training [87.58996020705258]
  Video Large Language Models (Video-LLMs) have recently shown strong derivation in basic video understanding tasks.<n>Video-LLMs struggle with compositional reasoning that requires multi-step explicit-temporal inference across object relations, interactions and events.<n>We propose STEP, a novel graph-guided self-training method that enables VideoLLMs to generate reasoning-rich finetuning data from any raw videos to improve itself.
  arXiv  Detail & Related papers  (2024-11-29T11:54:55Z)
• Understanding Chain-of-Thought in LLMs through Information Theory [16.78730663293352]
  We formalize Chain-of-Thought (CoT) reasoning in Large Language Models (LLMs) through an information-theoretic lens.<n>Specifically, our framework quantifies the information-gain' at each reasoning step, enabling the identification of failure modes.<n>We demonstrate the efficacy of our approach through extensive experiments on toy arithmetic, GSM8K and PRM800k datasets.
  arXiv  Detail & Related papers  (2024-11-18T19:14:36Z)
• ConMe: Rethinking Evaluation of Compositional Reasoning for Modern VLMs [95.15814662348245]
  Compositional Reasoning (CR) entails grasping the significance of attributes, relations, and word order. Recent Vision-Language Models (VLMs) have demonstrated remarkable proficiency in such reasoning tasks.
  arXiv  Detail & Related papers  (2024-06-12T12:54:27Z)
• Sparsity-Guided Holistic Explanation for LLMs with Interpretable Inference-Time Intervention [53.896974148579346]
  Large Language Models (LLMs) have achieved unprecedented breakthroughs in various natural language processing domains. The enigmatic black-box'' nature of LLMs remains a significant challenge for interpretability, hampering transparent and accountable applications. We propose a novel methodology anchored in sparsity-guided techniques, aiming to provide a holistic interpretation of LLMs.
  arXiv  Detail & Related papers  (2023-12-22T19:55:58Z)
• Counterfactual Explanations Using Optimization With Constraint Learning [0.0]
  We propose a generic and flexible approach to counterfactual explanations using optimization with constraint learning (CE-OCL) Specifically, we discuss how we can leverage an optimization with constraint learning framework for the generation of counterfactual explanations. We also propose two novel modeling approaches to address data manifold closeness and diversity, which are two key criteria for practical counterfactual explanations.
  arXiv  Detail & Related papers  (2022-09-22T13:27:21Z)
• Multilingual Multi-Aspect Explainability Analyses on Machine Reading Comprehension Models [76.48370548802464]
  This paper focuses on conducting a series of analytical experiments to examine the relations between the multi-head self-attention and the final MRC system performance. We discover that passage-to-question and passage understanding attentions are the most important ones in the question answering process. Through comprehensive visualizations and case studies, we also observe several general findings on the attention maps, which can be helpful to understand how these models solve the questions.
  arXiv  Detail & Related papers  (2021-08-26T04:23:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.