Ego-Human Motion Prediction with 3D-Aware LLM

• URL: http://arxiv.org/abs/2607.07001v1
• Date: Wed, 08 Jul 2026 04:51:24 GMT
• Title: Ego-Human Motion Prediction with 3D-Aware LLM
• Abstract summary: Ego3DLM is a tool for predicting human motion from an egocentric perspective.<n>It decodes past pose, future pose, past narration, and future narration in a single autoregressive pass.<n>Ego3DLM achieves state-of-the-art performance across future motion prediction, past motion tracking, and motion description.
• Score: 51.5064259568401
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Anticipating human motion from an egocentric perspective is fundamental for proactive assistance in AR/VR, human-robot collaboration, and embodied AI. While recent works incorporate language as a semantic prior to reduce the ill-posed nature of egocentric forecasting, they largely neglect the 3D spatial and semantic context that governs how motion unfolds, and treat pose and language prediction as separate inference streams. We introduce Ego3DLM, built on two core principles: accurate motion forecasting requires explicit spatial and semantic understanding of the 3D environment, and pose and language must be predicted holistically in a single pass, since motion is inherently tied to the semantic interpretation of actions being performed. Given three-point tracking, 3D scene features, and egocentric video, Ego3DLM simultaneously decodes past pose, future pose, past narration, and future narration in a single autoregressive pass, grounding predicted poses and descriptions in one another to enforce cross-modal and temporal consistency. We adopt a three-stage training scheme: (1) spatial-semantic scene awareness pretraining; (2) holistic instruction tuning over all four outputs in a single pass; and (3) GRPO-based reinforcement finetuning with intra- and inter-modal rewards that directly optimize pose-language fidelity. Experiments on the Nymeria benchmark demonstrate that Ego3DLM achieves state-of-the-art performance across future motion prediction, past motion tracking, and motion description, showing that 3D scene grounding and holistic cross-modal prediction yield physically plausible and semantically coherent motion forecasts. The project page is available at https://jaewoo97.github.io/Ego3DLM/.

Related papers

• GeomVLA: Unifying Scene, Motion, and Action in 3D [19.277341680583834]
  We present GeomVLA, a model that unifies perception, latent scene motion prediction, and action generation within a shared robot-centric 3D coordinate frame.<n>Our approach lifts pretrained VLM features into spatially grounded 3D scene tokens using depth and camera calibration.<n>We introduce a 3D Scene Trajectory Denoiser, a task-conditioned module that learns a latent representation of how scene points are expected to move in 3D.
  arXiv  Detail & Related papers  (2026-09-12T08:51:52Z)
• From Where to How: Continuous 4D Interaction Forecasting from Egocentric Video [62.898774619819676]
  Egocentric 4D interaction forecasting aims to anticipate both where future interactions will occur in 3D and how the human body will move to realize them.<n>We introduce Coherent4D, a large-scale egocentric dataset for continuous 4D interaction forecasting, comprising approximately 233K samples across three domains.<n>We propose HIGFlow, a Hand Interaction Guided Residual Flow framework that models forecasting as a cascaded where-to-how process.
  arXiv  Detail & Related papers  (2026-09-08T12:06:23Z)
• EgoMotion: Hierarchical Reasoning and Diffusion for Egocentric Vision-Language Motion Generation [74.07852250099559]
  We investigate Egocentric Vision-Language (Ego-VL) motion generation.<n>This task requires synthesizing 3D human motion conditioned jointly on first-person visual observations and natural language instructions.<n>We propose a hierarchical generative framework textbfEgoMotion to address this challenge.
  arXiv  Detail & Related papers  (2026-04-21T05:31:06Z)
• Flowing from Reasoning to Motion: Learning 3D Hand Trajectory Prediction from Egocentric Human Interaction Videos [42.207282959798]
  We present a large-scale egocentric dataset for interaction stage-aware 3D hand trajectory prediction with 219K 6DoF trajectories and 3M structured QA pairs for semantic, spatial, and motion reasoning.<n>We then introduce the EgoMAN model, a reasoning-to-motion framework that links vision-token reasoning and motion generation via a trajectory-language interface.
  arXiv  Detail & Related papers  (2025-12-18T18:59:01Z)
• Gaze Beyond the Frame: Forecasting Egocentric 3D Visual Span [50.60002620855774]
  We propose EgoSpanLift, a method that transforms egocentric visual span forecasting from 2D image planes to 3D scenes.<n>We also curate a benchmark from raw egocentric data, creating a testbed with 364.6K samples for 3D visual span forecasting.
  arXiv  Detail & Related papers  (2025-11-23T14:37:11Z)
• DreamPose3D: Hallucinative Diffusion with Prompt Learning for 3D Human Pose Estimation [18.240580699213197]
  We introduce DreamPose3D, a diffusion-based framework that combines action-aware reasoning with temporal imagination for 3D pose estimation.<n>DreamPose3D dynamically conditions the denoising process using task-relevant action prompts extracted from 2D pose sequences, capturing high-level intent.<n>To model the structural relationships between joints effectively, we introduce a representation encoder that incorporates kinematic joint affinity into the attention mechanism.<n>A hallucinative pose decoder predicts temporally coherent 3D pose during training, simulating how humans mentally reconstruct motion trajectories to resolve ambiguity in perception.
  arXiv  Detail & Related papers  (2025-11-12T17:31:41Z)
• FIction: 4D Future Interaction Prediction from Video [63.37136159797888]
  We introduce FIction for 4D future interaction prediction from videos.<n>Given an input video of a human activity, the goal is to predict which objects at what 3D locations the person will interact with in the next time period.
  arXiv  Detail & Related papers  (2024-12-01T18:44:17Z)
• Multimodal Sense-Informed Prediction of 3D Human Motions [16.71099574742631]
  This work introduces a novel multi-modal sense-informed motion prediction approach, which conditions high-fidelity generation on two modal information. The gaze information is regarded as the human intention, and combined with both motion and scene features, we construct a ternary intention-aware attention to supervise the generation. On two real-world benchmarks, the proposed method achieves state-of-the-art performance both in 3D human pose and trajectory prediction.
  arXiv  Detail & Related papers  (2024-05-05T12:38:10Z)
• Social-Transmotion: Promptable Human Trajectory Prediction [65.80068316170613]
  Social-Transmotion is a generic Transformer-based model that exploits diverse and numerous visual cues to predict human behavior.<n>Our approach is validated on multiple datasets, including JTA, JRDB, Pedestrians and Cyclists in Road Traffic, and ETH-UCY.
  arXiv  Detail & Related papers  (2023-12-26T18:56:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.