Flowing from Reasoning to Motion: Learning 3D Hand Trajectory Prediction from Egocentric Human Interaction Videos

• URL: http://arxiv.org/abs/2512.16907v1
• Date: Thu, 18 Dec 2025 18:59:01 GMT
• Title: Flowing from Reasoning to Motion: Learning 3D Hand Trajectory Prediction from Egocentric Human Interaction Videos
• Abstract summary: We present a large-scale egocentric dataset for interaction stage-aware 3D hand trajectory prediction with 219K 6DoF trajectories and 3M structured QA pairs for semantic, spatial, and motion reasoning.<n>We then introduce the EgoMAN model, a reasoning-to-motion framework that links vision-token reasoning and motion generation via a trajectory-language interface.
• Score: 42.207282959798
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Prior works on 3D hand trajectory prediction are constrained by datasets that decouple motion from semantic supervision and by models that weakly link reasoning and action. To address these, we first present the EgoMAN dataset, a large-scale egocentric dataset for interaction stage-aware 3D hand trajectory prediction with 219K 6DoF trajectories and 3M structured QA pairs for semantic, spatial, and motion reasoning. We then introduce the EgoMAN model, a reasoning-to-motion framework that links vision-language reasoning and motion generation via a trajectory-token interface. Trained progressively to align reasoning with motion dynamics, our approach yields accurate and stage-aware trajectories with generalization across real-world scenes.

Related papers

• Ego-Human Motion Prediction with 3D-Aware LLM [51.5064259568401]
  Ego3DLM is a tool for predicting human motion from an egocentric perspective.<n>It decodes past pose, future pose, past narration, and future narration in a single autoregressive pass.<n>Ego3DLM achieves state-of-the-art performance across future motion prediction, past motion tracking, and motion description.
  arXiv  Detail & Related papers  (2026-07-08T04:51:24Z)
• VLK: Learning Humanoid Loco-Manipulation from Synthetic Interactions in Reconstructed Scenes [71.14444761145435]
  Perception-based humanoid loco-manipulation requires connecting egocentric observations and task instructions to whole-body motion.<n>No existing data source provides this complete at scale.<n>We address this bottleneck by generating vision-language-kinematics supervision synthetically in reconstructed scenes.
  arXiv  Detail & Related papers  (2026-06-29T17:59:55Z)
• MolmoMotion: Forecasting Point Trajectories in 3D with Language Instruction [48.93378286720386]
  MolmoMotion is a general motion forecasting model that supports both autoregressive coordinate prediction and flow-matching-based trajectory generation.<n>It accurately predicts diverse motion patterns with different language instructions, and significantly outperforms existing motion prediction baselines on PointMotionBench.<n>It improves training efficiency and generalization for robot manipulation, and its predicted trajectories provide effective motion guidance for generative models to synthesize videos with more realistic object motion.
  arXiv  Detail & Related papers  (2026-06-17T00:19:00Z)
• Controllable Egocentric Video Generation via Occlusion-Aware Sparse 3D Hand Joints [87.13154261503168]
  Motion-controllable video generation is crucial for egocentric applications in virtual reality and embodied AI.<n>Existing methods often struggle to achieve 3D-consistent fine-grained hand articulation.<n>We propose a novel framework that generates egocentric videos from a single reference frame.
  arXiv  Detail & Related papers  (2026-03-12T10:02:23Z)
• MeshMimic: Geometry-Aware Humanoid Motion Learning through 3D Scene Reconstruction [54.36564144414704]
  MeshMimic is an innovative framework that bridges 3D scene reconstruction and embodied intelligence to enable humanoid robots to learn coupled "motion-terrain" interactions directly from video.<n>By leveraging state-of-the-art 3D vision models, our framework precisely segments and reconstructs both human trajectories and the underlying 3D geometry of terrains and objects.
  arXiv  Detail & Related papers  (2026-02-17T17:09:45Z)
• CoopDiff: Anticipating 3D Human-object Interactions via Contact-consistent Decoupled Diffusion [62.93198247045824]
  3D human-object interaction (HOI) anticipation aims to predict the future motion of humans and their manipulated objects, conditioned on the historical context.<n>We propose a novel contact-consistent decoupled diffusion framework CoopDiff, which employs two distinct branches to decouple human and object motion modeling.
  arXiv  Detail & Related papers  (2025-08-10T03:29:17Z)
• UPTor: Unified 3D Human Pose Dynamics and Trajectory Prediction for Human-Robot Interaction [0.688204255655161]
  We propose a technique to predict full-body pose and trajectory key-points in a global coordinate frame.<n>We use an off-the-shelf 3D human pose estimation module, a graph attention network, and a compact, non-autoregressive transformer.<n>In comparison to prior work, we show that our approach is compact, real-time, and accurate in predicting human navigation motion across all datasets.
  arXiv  Detail & Related papers  (2025-05-20T19:57:25Z)
• MADiff: Motion-Aware Mamba Diffusion Models for Hand Trajectory Prediction on Egocentric Videos [27.766405152248055]
  Hand trajectory prediction plays a vital role in comprehending human motion patterns. However, capturing high-level human intentions consistent with reasonable temporal causality is challenging when only egocentric videos are available. We propose a novel hand trajectory prediction method dubbed MADiff, which forecasts future hand waypoints with diffusion models.
  arXiv  Detail & Related papers  (2024-09-04T12:06:33Z)
• Past Movements-Guided Motion Representation Learning for Human Motion Prediction [0.0]
  We propose a self-supervised learning framework designed to enhance motion representation. The framework consists of two stages: first, the network is pretrained through the self-reconstruction of past sequences, and the guided reconstruction of future sequences based on past movements. Our method reduces the average prediction errors by 8.8% across Human3.6, 3DPW, and AMASS datasets.
  arXiv  Detail & Related papers  (2024-08-04T17:00:37Z)
• Diff-IP2D: Diffusion-Based Hand-Object Interaction Prediction on Egocentric Videos [22.81433371521832]
  We propose Diff-IP2D to forecast future hand trajectories and object affordances concurrently in an iterative non-autoregressive manner. Our method significantly outperforms the state-of-the-art baselines on both the off-the-shelf metrics and our newly proposed evaluation protocol.
  arXiv  Detail & Related papers  (2024-05-07T14:51:05Z)
• Social-Transmotion: Promptable Human Trajectory Prediction [65.80068316170613]
  Social-Transmotion is a generic Transformer-based model that exploits diverse and numerous visual cues to predict human behavior.<n>Our approach is validated on multiple datasets, including JTA, JRDB, Pedestrians and Cyclists in Road Traffic, and ETH-UCY.
  arXiv  Detail & Related papers  (2023-12-26T18:56:49Z)
• Uncertainty-aware State Space Transformer for Egocentric 3D Hand Trajectory Forecasting [79.34357055254239]
  Hand trajectory forecasting is crucial for enabling a prompt understanding of human intentions when interacting with AR/VR systems. Existing methods handle this problem in a 2D image space which is inadequate for 3D real-world applications. We set up an egocentric 3D hand trajectory forecasting task that aims to predict hand trajectories in a 3D space from early observed RGB videos in a first-person view.
  arXiv  Detail & Related papers  (2023-07-17T04:55:02Z)
• GraMMaR: Ground-aware Motion Model for 3D Human Motion Reconstruction [61.833152949826946]
  We propose a novel Ground-aware Motion Model for 3D Human Motion Reconstruction, named GraMMaR. GraMMaR learns the distribution of transitions in both pose and interaction between every joint and ground plane at each time step of a motion sequence. It is trained to explicitly promote consistency between the motion and distance change towards the ground.
  arXiv  Detail & Related papers  (2023-06-29T07:22:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.