EgoMotion: Hierarchical Reasoning and Diffusion for Egocentric Vision-Language Motion Generation

• URL: http://arxiv.org/abs/2604.19105v1
• Date: Tue, 21 Apr 2026 05:31:06 GMT
• Title: EgoMotion: Hierarchical Reasoning and Diffusion for Egocentric Vision-Language Motion Generation
• Abstract summary: We investigate Egocentric Vision-Language (Ego-VL) motion generation.<n>This task requires synthesizing 3D human motion conditioned jointly on first-person visual observations and natural language instructions.<n>We propose a hierarchical generative framework textbfEgoMotion to address this challenge.
• Score: 74.07852250099559
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Faithfully modeling human behavior in dynamic environments is a foundational challenge for embodied intelligence. While conditional motion synthesis has achieved significant advances, egocentric motion generation remains largely underexplored due to the inherent complexity of first-person perception. In this work, we investigate Egocentric Vision-Language (Ego-VL) motion generation. This task requires synthesizing 3D human motion conditioned jointly on first-person visual observations and natural language instructions. We identify a critical \textit{reasoning-generation entanglement} challenge: the simultaneous optimization of semantic reasoning and kinematic modeling introduces gradient conflicts. These conflicts systematically degrade the fidelity of multimodal grounding and motion quality. To address this challenge, we propose a hierarchical generative framework \textbf{EgoMotion}. Inspired by the biological decoupling of cognitive reasoning and motor control, EgoMotion operates in two stages. In the Cognitive Reasoning stage, A vision-language model (VLM) projects multimodal inputs into a structured space of discrete motion primitives. This forces the VLM to acquire goal-consistent representations, effectively bridging the semantic gap between high-level perceptual understanding and low-level action execution. In the Motion Generation stage, these learned representations serve as expressive conditioning signals for a diffusion-based motion generator. By performing iterative denoising within a continuous latent space, the generator synthesizes physically plausible and temporally coherent trajectories. Extensive evaluations demonstrate that EgoMotion achieves state-of-the-art performance, and produces motion sequences that are both semantically grounded and kinematically superior to existing approaches.

Related papers

• Ego-Human Motion Prediction with 3D-Aware LLM [51.5064259568401]
  Ego3DLM is a tool for predicting human motion from an egocentric perspective.<n>It decodes past pose, future pose, past narration, and future narration in a single autoregressive pass.<n>Ego3DLM achieves state-of-the-art performance across future motion prediction, past motion tracking, and motion description.
  arXiv  Detail & Related papers  (2026-07-08T04:51:24Z)
• MotiMotion: Motion-Controlled Video Generation with Visual Reasoning [49.243234645532745]
  We introduce MotiMotion, a framework that reformulates motion control as a reasoning-then-generation problem.<n>To encourage causally grounded and commonsense-consistent interactions, we leverage a training-free vision-language reasoner.<n>To further improve motion naturalness, we propose a confidence-aware control scheme that modulates guidance strength.
  arXiv  Detail & Related papers  (2026-05-21T17:59:36Z)
• EggHand: A Multimodal Foundation Model for Egocentric Hand Pose Forecasting [11.687067206676625]
  EggHand is a foundation-model-based framework for egocentric hand pose forecasting.<n>It unifies multimodal semantic reasoning with dynamic motion modeling.<n>Our approach overcomes the brittleness of generic visual encoders under ego-motion.
  arXiv  Detail & Related papers  (2026-05-08T12:09:27Z)
• Ego-InBetween: Generating Object State Transitions in Ego-Centric Videos [56.20829168540647]
  EgoIn is a framework that infers the multi-step transition process between two given states using TransitionVLM.<n>It generates a sequence of frames based on transition conditions produced by the proposed Transition Conditioning module.<n>Experiments on human-object and robot-object interaction datasets demonstrate EgoIn's superior performance in generating semantically meaningful and visually coherent transformation sequences.
  arXiv  Detail & Related papers  (2026-04-20T03:07:22Z)
• HO-Flow: Generalizable Hand-Object Interaction Generation with Latent Flow Matching [113.81911881001905]
  HO-Flow is a framework for synthesizing realistic hand-object motion sequences from texts and canoncial 3D objects.<n>It first employs an interaction-aware variational autoencoder to encode sequences of hand and object motions into a unified latent manifold.<n>It then leverages a masked flow matching model that combines auto-regressive temporal reasoning with continuous latent generation.
  arXiv  Detail & Related papers  (2026-04-12T22:06:11Z)
• Coordinate-Based Dual-Constrained Autoregressive Motion Generation [28.415421434583482]
  We propose a flexible, high-fidelity, and semantically faithful text-to-motion framework, named Coordinate-based Dual-constrained Autoregressive Motion Generation (CDAMD)<n>With motion coordinates as input, CDAMD follows the autoregressive paradigm and leverages diffusion-inspired multi-layer perceptrons to enhance the fidelity of predicted motions.<n>Since there is limited work on coordinate-based motion synthesis, we establish new benchmarks for both text-to-motion generation and motion editing.
  arXiv  Detail & Related papers  (2026-04-09T11:05:29Z)
• EgoForge: Goal-Directed Egocentric World Simulator [26.712565464146937]
  EgoForge is a goal-directed world simulator that generates coherent, first-person video rollouts from minimal static inputs.<n>VideoDiffusionNFT is a trajectory-level reward-guided refinement that optimize goal completion, temporal causality, scene consistency, and perceptual fidelity during diffusion sampling.
  arXiv  Detail & Related papers  (2026-03-20T17:46:55Z)
• Hand-Aware Egocentric Motion Reconstruction with Sequence-Level Context [17.735273173582716]
  We present HaMoS, the first hand-aware, sequence-level diffusion framework that directly conditions on both head trajectory and intermittently visible hand cues.<n>We also demonstrate that sequence-level contexts such as body shape and field-of-view are crucial for accurate motion reconstruction.
  arXiv  Detail & Related papers  (2025-12-22T11:26:41Z)
• EchoMotion: Unified Human Video and Motion Generation via Dual-Modality Diffusion Transformer [64.69014756863331]
  We introduce EchoMotion, a framework designed to model the joint distribution of appearance and human motion.<n>We also propose MVS-RoPE, which offers unified 3D positional encoding for both video and motion tokens.<n>Our findings reveal that explicitly representing human motion is to appearance, significantly boosting the coherence and plausibility of human-centric video generation.
  arXiv  Detail & Related papers  (2025-12-21T17:08:14Z)
• UniEgoMotion: A Unified Model for Egocentric Motion Reconstruction, Forecasting, and Generation [26.03507721111338]
  We introduce Egocentric Motion Generation and Egocentric Motion Forecasting, two novel tasks that utilize first-person images for scene-aware motion synthesis.<n>We propose UniEgoMotion, a unified conditional motion diffusion model with a novel head-centric motion representation tailored for egocentric devices.<n>UniEgoMotion achieves state-of-the-art performance in egocentric motion reconstruction and is the first to generate motion from a single egocentric image.
  arXiv  Detail & Related papers  (2025-08-02T00:41:20Z)
• MotionGPT3: Human Motion as a Second Modality [28.616340011811843]
  MotionGPT3 is a bimodal motion-language model for both understanding and generation.<n>A dual-stream Transformer with shared attention preserves modality-specific routes while enabling controlled, bidirectional information flow.<n>Experiments show that MotionGPT3 achieves 2x faster convergence in training loss and up to 4x faster convergence in validation.
  arXiv  Detail & Related papers  (2025-06-30T17:42:22Z)
• InterDyn: Controllable Interactive Dynamics with Video Diffusion Models [50.38647583839384]
  We propose InterDyn, a framework that generates videos of interactive dynamics given an initial frame and a control signal encoding the motion of a driving object or actor.<n>Our key insight is that large video generation models can act as both neurals and implicit physics simulators'', having learned interactive dynamics from large-scale video data.
  arXiv  Detail & Related papers  (2024-12-16T13:57:02Z)
• Priority-Centric Human Motion Generation in Discrete Latent Space [59.401128190423535]
  We introduce a Priority-Centric Motion Discrete Diffusion Model (M2DM) for text-to-motion generation. M2DM incorporates a global self-attention mechanism and a regularization term to counteract code collapse. We also present a motion discrete diffusion model that employs an innovative noise schedule, determined by the significance of each motion token.
  arXiv  Detail & Related papers  (2023-08-28T10:40:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.