Egocentric World Model for Photorealistic Hand-Object Interaction Synthesis

• URL: http://arxiv.org/abs/2603.13615v1
• Date: Fri, 13 Mar 2026 21:46:17 GMT
• Title: Egocentric World Model for Photorealistic Hand-Object Interaction Synthesis
• Abstract summary: EgoHOI is an egocentric HOI world model that breaks away from this shortcut to simulate photorealistic, contact-consistent interactions from action signals alone.<n>EgoHOI distills geometric and kinematic priors from 3D estimates into physics-informed embeddings.<n>Experiments on the HOT3D dataset demonstrate consistent gains over strong baselines, and ablations validate the effectiveness of our design.
• Score: 25.153233500074194
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: To serve as a scalable data source for embodied AI, world models should act as true simulators that infer interaction dynamics strictly from user actions, rather than mere conditional video generators relying on privileged future object states. In this context, egocentric Human-Object Interaction (HOI) world models are critical for predicting physically grounded first-person rollouts. However, building such models is profoundly challenging due to rapid head motions, severe occlusions, and high-DoF hand articulations that abruptly alter contact topologies. Consequently, existing approaches often circumvent these physics challenges by resorting to conditional video generation with access to known future object trajectories. We introduce EgoHOI, an egocentric HOI world model that breaks away from this shortcut to simulate photorealistic, contact-consistent interactions from action signals alone. To ensure physical accuracy without future-state inputs, EgoHOI distills geometric and kinematic priors from 3D estimates into physics-informed embeddings. These embeddings regularize the egocentric rollouts toward physically valid dynamics. Experiments on the HOT3D dataset demonstrate consistent gains over strong baselines, and ablations validate the effectiveness of our physics-informed design.

Related papers

• GAUGE: A Measurement-Grounded Benchmark for Physical Fidelity in Simulation Engines and Video World Models [66.77745044586173]
  GAUGE is a real-world-grounded diagnostic benchmark for evaluating how numerical simulators and generative video world models reproduce or deviate from real-world physics.<n>We benchmark Isaac Sim, Genesis, and Newton on 14 task families using generalized trajectory errors, and evaluate 6 image-to-video models on 5 rigid-body tasks.<n>Our results reveal no uniformly faithful physics engine, with the largest discrepancies arising in impulsive contact, rapid textile motion, and volumetric deformation.
  arXiv  Detail & Related papers  (2026-08-06T12:19:38Z)
• A Physics-Grounded Benchmark for Multi-Agent Dynamics in World Models [96.64934195520802]
  We introduce CrashTwin, a physics-grounded evaluation framework designed to stress-test the physical trustworthiness of world models.<n>CrashTwin couples a diverse dataset of multi-agent collision scenarios, comprising 25K controllable synthetic 12K in-the-wild real-world collision sequences.<n>We propose a diagnostic suite that systematically evaluates three dimensions:-temporal consistency, momentum and kinetic energy conservation, and world-dynamics integrity.
  arXiv  Detail & Related papers  (2026-06-27T06:13:35Z)
• IOI: Decoupling Kinematics and Physics for Interactive World Models [46.3330122411516]
  We propose IOI, a hybrid interactive world model integrating kinematics with learned physical dynamics.<n> IOI introduces explicit kinematic guidance, computing kinematics sequences for accurate motion trajectories.<n> experiments on the RoboTwin benchmark validate IOI across kinematic fidelity, out-of-distribution, and policy evaluation.
  arXiv  Detail & Related papers  (2026-06-22T13:09:34Z)
• HO-Flow: Generalizable Hand-Object Interaction Generation with Latent Flow Matching [113.81911881001905]
  HO-Flow is a framework for synthesizing realistic hand-object motion sequences from texts and canoncial 3D objects.<n>It first employs an interaction-aware variational autoencoder to encode sequences of hand and object motions into a unified latent manifold.<n>It then leverages a masked flow matching model that combines auto-regressive temporal reasoning with continuous latent generation.
  arXiv  Detail & Related papers  (2026-04-12T22:06:11Z)
• EgoFlow: Gradient-Guided Flow Matching for Egocentric 6DoF Object Motion Generation [47.32597153743819]
  We present EgoFlow, a flow-matching framework that synthesizes realistic and physically plausible trajectories conditioned on multimodal egocentric observations.<n>Our results highlight the promise of flow-based generative modeling for scalable and physically grounded egocentric motion understanding.
  arXiv  Detail & Related papers  (2026-04-01T21:43:57Z)
• EgoGrasp: World-Space Hand-Object Interaction Estimation from Egocentric Videos [25.047225764745978]
  We propose EgoGrasp, the first method to reconstruct world-space hand-object interactions (W-HOI) from egocentric monocular videos with dynamic cameras in the wild.<n>In experiments, we prove our method achieving state-of-the-art performance in W-HOI reconstruction.
  arXiv  Detail & Related papers  (2026-01-03T03:08:48Z)
• SpriteHand: Real-Time Versatile Hand-Object Interaction with Autoregressive Video Generation [64.3409486422946]
  We present SpriteHand, an autoregressive video generation framework for real-time synthesis of hand-object interaction videos.<n>Our model employs a causal inference architecture for autoregressive generation and leverages a hybrid post-training approach to enhance visual realism and temporal coherence.<n> Experiments demonstrate superior visual quality, physical plausibility, and interaction fidelity compared to both generative and engine-based baselines.
  arXiv  Detail & Related papers  (2025-12-01T18:13:40Z)
• What Happens Next? Anticipating Future Motion by Generating Point Trajectories [76.16266402727643]
  We consider the problem of forecasting motion from a single image, predicting how objects in the world are likely to move.<n>We formulate this task as conditional generation of dense trajectory grids with a model that closely follows the architecture of modern video generators.<n>This approach captures scene-wide dynamics and uncertainty, yielding more accurate and diverse predictions than prior regressors and generators.
  arXiv  Detail & Related papers  (2025-09-25T21:03:56Z)
• SimGenHOI: Physically Realistic Whole-Body Humanoid-Object Interaction via Generative Modeling and Reinforcement Learning [6.255814224573073]
  SimGenHOI is a unified framework that combines the strengths of generative modeling and reinforcement learning to produce controllable and physically plausible HOI.<n>Our HOI generative model, based on Diffusion Transformers (DiT), predicts a set of key actions conditioned on text prompts, object geometry, sparse object waypoints, and the initial humanoid pose.<n>To ensure physical realism, we design a contact-aware whole-body control policy trained with reinforcement learning, which tracks the generated motions while correcting artifacts such as penetration and foot sliding.
  arXiv  Detail & Related papers  (2025-08-18T15:20:46Z)
• InterDyn: Controllable Interactive Dynamics with Video Diffusion Models [50.38647583839384]
  We propose InterDyn, a framework that generates videos of interactive dynamics given an initial frame and a control signal encoding the motion of a driving object or actor.<n>Our key insight is that large video generation models can act as both neurals and implicit physics simulators'', having learned interactive dynamics from large-scale video data.
  arXiv  Detail & Related papers  (2024-12-16T13:57:02Z)
• PhysDreamer: Physics-Based Interaction with 3D Objects via Video Generation [62.53760963292465]
  PhysDreamer is a physics-based approach that endows static 3D objects with interactive dynamics. We present our approach on diverse examples of elastic objects and evaluate the realism of the synthesized interactions through a user study.
  arXiv  Detail & Related papers  (2024-04-19T17:41:05Z)
• Hindsight for Foresight: Unsupervised Structured Dynamics Models from Physical Interaction [24.72947291987545]
  Key challenge for an agent learning to interact with the world is to reason about physical properties of objects. We propose a novel approach for modeling the dynamics of a robot's interactions directly from unlabeled 3D point clouds and images.
  arXiv  Detail & Related papers  (2020-08-02T11:04:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.