HumanMoveVQA: Can Video MLLMs reason about human movement in videos?
- URL: http://arxiv.org/abs/2606.27999v2
- Date: Mon, 29 Jun 2026 10:15:53 GMT
- Title: HumanMoveVQA: Can Video MLLMs reason about human movement in videos?
- Abstract summary: HumanMoveVQA is the first benchmark designed to evaluate global trajectory and orientation reasoning from an exocentric perspective.<n>We propose a scalable, multi-stage pipeline that lifts 2D video observations into world-consistent 3D motion tracks.<n>Our evaluation reveals a critical capability gap in state-of-the-art proprietary models on deep human motion understanding.
- Score: 21.158494377012286
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Despite the rapid advance of Multimodal Large Language Models (MLLMs) in high-level video understanding, a fundamental bottleneck remains: these models collapse complex human motion into coarse semantic labels. Existing benchmarks mostly focus on scene-centric events or local joint articulations, failing to probe global human motion in space over time (trajectory and orientation changes). We introduce HumanMoveVQA, the first comprehensive benchmark designed to evaluate global trajectory and orientation reasoning from an exocentric perspective. Our benchmark utilizes a first-frame anchored world coordinate system, preserving translation and rotation relative to a fixed starting point. We propose a scalable, multi-stage pipeline that lifts 2D video observations into world-consistent 3D motion tracks to generate over 10K structured question-answer pairs across seven reasoning categories, including motion aggregation, sequential ordering, and trajectory-level inference. Our extensive evaluation reveals a critical capability gap in state-of-the-art proprietary models on deep human motion understanding. However, we demonstrate that this is a learnable problem; by fine-tuning an open-source baseline with our targeted, world-consistent supervision, we achieve a significant improvement. HumanMoveVQA establishes a rigorous geometric foundation for developing next-generation, movement-aware video understanding models.
Related papers
- HiPHI: A Large-Scale Benchmark for High-Precision Human Motion and Object-Interaction [48.4618478154232]
HiPHI is a 600+ hour scale high-fidelity whole-body human motion dataset.<n>It provides sub-millimeter spatial marker tracking accuracy for full-body human motion and mesh-level object trajectories.<n>We introduce a benchmark suite evaluating motion-space diversity, interaction grounding, object consistency, and physical AI applications.
arXiv Detail & Related papers (2026-08-17T07:54:26Z) - ACE-Data-0: Human-Centric Ambient Capture as Embodied Data Engine [83.02804728003862]
Ambient Capture Engine (ACE) transforms real home environments into spatially, temporally synchronized recording studios.<n>ACE records egocentric and multi-view exocentric video, full-body and articulated hand motion, object geometry and 6-DoF trajectories, audio, and signals as a unified multisensory stream.<n>Dataset spans atomic manipulation, long-horizon chains of household activities, and human-scene interaction, while preserving natural behavioral variation through goal-level rather than step-by-step instructions.
arXiv Detail & Related papers (2026-07-30T17:59:50Z) - Directing the World: Fast Autoregressive Video Generation with Compositional Human-Camera Control [56.828569670698975]
"Directing the World" is a fast autoregressive framework for controllable world-model video generation with compositional human-motion and camera-trajectory control.<n>Our key idea is to decouple control learning while preserving a unified autoregressive video prior.<n>For human motion control, we design a t-guided Dynamic Projection mechanism and a refined Motion-CFG strategy.
arXiv Detail & Related papers (2026-06-26T11:08:09Z) - EgoMotion: Hierarchical Reasoning and Diffusion for Egocentric Vision-Language Motion Generation [74.07852250099559]
We investigate Egocentric Vision-Language (Ego-VL) motion generation.<n>This task requires synthesizing 3D human motion conditioned jointly on first-person visual observations and natural language instructions.<n>We propose a hierarchical generative framework textbfEgoMotion to address this challenge.
arXiv Detail & Related papers (2026-04-21T05:31:06Z) - Superman: Unifying Skeleton and Vision for Human Motion Perception and Generation [32.57062686780495]
Superman is a unified framework that bridges visual perception with temporal, skeleton-based motion generation.<n>This module flexibly processes diverse, temporal inputs, unifying 3D skeleton pose estimation from video (perception) with skeleton-based motion prediction and in-betweening (generation)
arXiv Detail & Related papers (2026-02-02T17:59:01Z) - LatentMove: Towards Complex Human Movement Video Generation [35.83863053692456]
We present LatentMove, a DiT-based framework specifically tailored for highly dynamic human animation.<n>Our architecture incorporates a conditional control branch and learnable face/body tokens to preserve consistency as well as fine-grained details across frames.<n>We introduce Complex-Human-Videos (CHV), a dataset featuring diverse, challenging human motions designed to benchmark the robustness of I2V systems.
arXiv Detail & Related papers (2025-05-28T07:10:49Z) - Mocap-2-to-3: Multi-view Lifting for Monocular Motion Recovery with 2D Pretraining [49.223455189395025]
Mocap-2-to-3 is a novel framework that performs multi-view lifting from monocular input.<n>To leverage abundant 2D data, we decompose complex 3D motion into multi-view syntheses.<n>Our method surpasses state-of-the-art approaches in both camera-space motion realism and world-grounded human positioning.
arXiv Detail & Related papers (2025-03-05T06:32:49Z) - Universal Humanoid Motion Representations for Physics-Based Control [71.46142106079292]
We present a universal motion representation that encompasses a comprehensive range of motor skills for physics-based humanoid control.
We first learn a motion imitator that can imitate all of human motion from a large, unstructured motion dataset.
We then create our motion representation by distilling skills directly from the imitator.
arXiv Detail & Related papers (2023-10-06T20:48:43Z) - MotionBERT: A Unified Perspective on Learning Human Motion
Representations [46.67364057245364]
We present a unified perspective on tackling various human-centric video tasks by learning human motion representations from large-scale and heterogeneous data resources.
We propose a pretraining stage in which a motion encoder is trained to recover the underlying 3D motion from noisy partial 2D observations.
We implement motion encoder with a Dual-stream Spatio-temporal Transformer (DSTformer) neural network.
arXiv Detail & Related papers (2022-10-12T19:46:25Z) - GLAMR: Global Occlusion-Aware Human Mesh Recovery with Dynamic Cameras [99.07219478953982]
We present an approach for 3D global human mesh recovery from monocular videos recorded with dynamic cameras.
We first propose a deep generative motion infiller, which autoregressively infills the body motions of occluded humans based on visible motions.
In contrast to prior work, our approach reconstructs human meshes in consistent global coordinates even with dynamic cameras.
arXiv Detail & Related papers (2021-12-02T18:59:54Z) - Task-Generic Hierarchical Human Motion Prior using VAEs [44.356707509079044]
A deep generative model that describes human motions can benefit a wide range of fundamental computer vision and graphics tasks.
We present a method for learning complex human motions independent of specific tasks using a combined global and local latent space.
We demonstrate the effectiveness of our hierarchical motion variational autoencoder in a variety of tasks including video-based human pose estimation.
arXiv Detail & Related papers (2021-06-07T23:11:42Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.