NextMotionQA: Benchmarking and Judging Human Motion Understanding with Vision-Language Models

• URL: http://arxiv.org/abs/2606.04773v1
• Date: Wed, 03 Jun 2026 11:53:57 GMT
• Title: NextMotionQA: Benchmarking and Judging Human Motion Understanding with Vision-Language Models
• Abstract summary: NextMotionQA is a comprehensive benchmark that leverages vision-language models for expert-verified dataset.<n>NextMotionQA features three complementary tasks: multiple-choice question answering, video captioning, and fine-grained error correction.<n>Our evaluation of twelve representative VLMs uncovers critical capability gaps and weakness that remain invisible under conventional, single-task evaluations.
• Score: 41.84042766842064
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reliable evaluation of human motion understanding is fundamental to advancing embodied AI, robotics, and animation. However, existing benchmarks suffer from coarse semantic granularity, undifferentiated difficulty, limited annotation quality, and pervasive answer ambiguity, leaving them unable to diagnose where current models fail. To bridge this gap, we introduce NextMotionQA, a comprehensive benchmark that leverages vision-language models (VLMs) for semi-automated, expert-verified dataset. NextMotionQA features three complementary tasks: multiple-choice question answering, video captioning, and fine-grained error correction. Each task is systematically structured across three core semantic axes and stratified into three task complexity levels. Our extensive evaluation of twelve representative VLMs uncovers critical capability gaps and weakness that remain invisible under conventional, single-task evaluations. In a complementary direction, recent work has begun using VLMs as judges for text-to-motion evaluation; we ask whether they show the same degradation under harder tasks. We find that VLMs align strongly with expert ratings on coarse criteria (Cohen's κ=0.70) but break down on fine-grained, part-level judgment (κ=0.10), validating the paradigm in its strong regime while clarifying its limits.

Related papers

• Beyond Shortcuts: Mitigating Visual Illusions in Frozen VLMs via Qualitative Reasoning [14.700305370711973]
  We propose a training-free, data-centric framework designed to fortify visual grounding in frozen Vision-Language Models.<n>Our framework was evaluated on the DataCV 2026 Challenge (Task I: Classic Illusion Understanding), where it ranked 2nd place overall.<n>Our success underscores the potential of structured qualitative grounding as a robust paradigm for developing next-generation, illusion-resistant vision-language systems.
  arXiv  Detail & Related papers  (2026-04-29T03:12:29Z)
• ReVSI: Rebuilding Visual Spatial Intelligence Evaluation for Accurate Assessment of VLM 3D Reasoning [59.558706734431276]
  Current evaluations of spatial intelligence can be systematically invalid under modern vision-language model (VLM) settings.<n>We introduce ReVSI, a benchmark and protocol that ensures each QA pair is answerable and correct under the model's actual inputs.
  arXiv  Detail & Related papers  (2026-04-27T10:45:51Z)
• Benchmarking at the Edge of Comprehension [38.43582342860192]
  If benchmarking becomes infeasible, our ability to measure any progress in AI is at stake.<n>We propose Critique-Resilient Benchmarking, an adversarial framework designed to compare models even when full human understanding is infeasible.<n>Unlike standard benchmarking, humans serve as bounded verifiers and focus on localized claims.
  arXiv  Detail & Related papers  (2026-02-15T20:51:29Z)
• LIBERO-X: Robustness Litmus for Vision-Language-Action Models [32.29541801424534]
  This work systematically rethinks VLA benchmarking from both evaluation and data perspectives.<n>We introduce LIBERO-X, a benchmark featuring a hierarchical evaluation protocol with progressive difficulty levels targeting three core capabilities.<n> Experiments with representative VLA models reveal significant performance drops under cumulative perturbations.
  arXiv  Detail & Related papers  (2026-02-06T09:59:12Z)
• CounterVQA: Evaluating and Improving Counterfactual Reasoning in Vision-Language Models for Video Understanding [13.628041236679229]
  Vision Language Models (VLMs) have recently shown significant advancements in video understanding, but their capability for counterfactual reasoning remains underexplored.<n>We introduce CounterVQA, a video-based benchmark featuring three progressive difficulty levels that assess different aspects of counterfactual reasoning.<n>We develop a post-training method, CFGPT, that enhances a model's visual counterfactual reasoning ability by distilling its counterfactual reasoning capability from the language modality.
  arXiv  Detail & Related papers  (2025-11-25T04:59:55Z)
• Q-Doc: Benchmarking Document Image Quality Assessment Capabilities in Multi-modal Large Language Models [19.598563198222035]
  We propose Q-Doc to systematically probing DIQA capabilities of MLLMs at coarse, middle, and fine granularity levels.<n>We show that while MLLMs possess nascent DIQA abilities, they exhibit critical limitations: inconsistent scoring, distortion misidentification, and severity misjudgment.<n>Our work provides a benchmark for DIQA capabilities in MLLMs, revealing pronounced deficiencies in their quality perception and promising pathways for enhancement.
  arXiv  Detail & Related papers  (2025-11-14T15:41:17Z)
• Plug-and-Play Clarifier: A Zero-Shot Multimodal Framework for Egocentric Intent Disambiguation [60.63465682731118]
  The performance of egocentric AI agents is fundamentally limited by multimodal intent ambiguity.<n>We introduce the Plug-and-Play Clarifier, a zero-shot and modular framework that decomposes the problem into discrete, solvable sub-tasks.<n>Our framework improves the intent clarification performance of small language models by approximately 30%, making them competitive with significantly larger counterparts.
  arXiv  Detail & Related papers  (2025-11-12T04:28:14Z)
• Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgetting [70.83781268763215]
  Vision-language models (VLMs) have achieved impressive performance across diverse multimodal tasks by leveraging large-scale pre-training.<n>VLMs face unique challenges such as cross-modal feature drift, parameter interference due to shared architectures, and zero-shot capability erosion.<n>This survey aims to serve as a comprehensive and diagnostic reference for researchers developing lifelong vision-language systems.
  arXiv  Detail & Related papers  (2025-08-06T09:03:10Z)
• Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarks [229.73714829399802]
  This survey probes the core challenges that the rise of Large Language Models poses for evaluation.<n>We identify and analyze two pivotal transitions: (i) from task-specific to capability-based evaluation, which reorganizes benchmarks around core competencies such as knowledge, reasoning, instruction following, multi-modal understanding, and safety.<n>We will dissect this issue, along with the core challenges of the above two transitions, from the perspectives of methods, datasets, evaluators, and metrics.
  arXiv  Detail & Related papers  (2025-04-26T07:48:52Z)
• Video SimpleQA: Towards Factuality Evaluation in Large Video Language Models [77.96693360763925]
  Video SimpleQA is the first comprehensive benchmark tailored for factuality evaluation in video contexts.<n>Our work differs from existing video benchmarks through the following key features: Knowledge required: demanding integration of external knowledge beyond the video's explicit narrative.<n>Short-form definitive answer: Answers are crafted as unambiguous and definitively correct in a short format with minimal scoring variance.
  arXiv  Detail & Related papers  (2025-03-24T17:46:09Z)
• AutoBench-V: Can Large Vision-Language Models Benchmark Themselves? [65.92331309449015]
  We introduce AutoBench-V, an automated framework for serving evaluation on demand, i.e., benchmarking LVLMs based on specific aspects of model capability.<n>Through an extensive evaluation of nine popular LVLMs across five demanded user inputs, the framework shows effectiveness and reliability.
  arXiv  Detail & Related papers  (2024-10-28T17:55:08Z)
• Towards Robust Text-Prompted Semantic Criterion for In-the-Wild Video Quality Assessment [54.31355080688127]
  We introduce a text-prompted Semantic Affinity Quality Index (SAQI) and its localized version (SAQI-Local) using Contrastive Language-Image Pre-training (CLIP) BVQI-Local demonstrates unprecedented performance, surpassing existing zero-shot indices by at least 24% on all datasets. We conduct comprehensive analyses to investigate different quality concerns of distinct indices, demonstrating the effectiveness and rationality of our design.
  arXiv  Detail & Related papers  (2023-04-28T08:06:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.