Real2Sim in HOI: Toward Physically Plausible HOI Reconstruction from Monocular Videos

• URL: http://arxiv.org/abs/2605.14462v1
• Date: Thu, 14 May 2026 06:56:57 GMT
• Title: Real2Sim in HOI: Toward Physically Plausible HOI Reconstruction from Monocular Videos
• Abstract summary: HOI reconstruction should not stop at tracking a human and an object, but should recover the relation that makes their motion a coherent interaction.<n>We introduce $textbfHA-HOI$, a framework for reconstructing physically plausible 4D HOI animation from in-the-wild monocular videos.<n>Our work takes a step toward turning general monocular HOI videos into scalable demonstrations for humanoid-object behavior.
• Score: 26.750787853601413
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recovering 4D human-object interaction (HOI) from monocular video is a key step toward scalable 3D content creation, embodied AI, and simulation-based learning. Recent methods can reconstruct temporally coherent human and object trajectories, but these trajectories often remain visual artifacts while failing to preserve stable contact, functional manipulation, or physical plausibility when used as reference motions for humanoid-object simulation. This reveals a fundamental interaction gap: HOI reconstruction should not stop at tracking a human and an object, but should recover the relation that makes their motion a coherent interaction. We introduce $\textbf{HA-HOI}$, a framework for reconstructing physically plausible 4D HOI animation from in-the-wild monocular videos. Instead of treating the human and object as independent entities in an ambiguous monocular 3D space, we propose a $\textit{human-first, object-follow}$ formulation. The human motion is recovered as the interaction anchor, and the object is reconstructed, aligned, and refined relative to the human action. The resulting kinematic trajectory is then projected into a physics-based humanoid-object simulation, where it acts as a teacher trajectory for stable physical rollout. Across benchmark and in-the-wild videos, $\textbf{HA-HOI}$ improves human-object alignment, contact consistency, temporal stability, and simulation readiness over prior monocular HOI reconstruction methods. By moving beyond visually plausible trajectory recovery toward physically grounded interaction animation, our work takes a step toward turning general monocular HOI videos into scalable demonstrations for humanoid-object behavior. Project page: https://knoxzhao.github.io/real2sim_in_HOI/

Related papers

• GRAIL: Generating Humanoid Loco-Manipulation from 3D Assets and Video Priors [113.71148915419246]
  We present GRAIL, a digital generation pipeline that composes 3D assets, simulator-ready scenes, and priors from video foundation models (VFMs) to synthesize interactions without rebuilding physical environments or teleoperating the robot.<n>GRAIL starts from fully specified 3D configurations in which object geometry, camera parameters, metric scale, environment depth, and a robot-proportioned character are known before video generation and reused during reconstruction.<n>We retarget the recovered motions to a humanoid robot and train complementary task-general morphology trackers.<n>GRAIL produces over 20,000 sequences spanning pick-up, object manipulation, sitting
  arXiv  Detail & Related papers  (2026-06-03T17:57:45Z)
• PhyGenHOI: Physically-Aware 4D Generation of Dynamic Human-Object Interactions [9.803936920993705]
  We introduce PhyGenHOI, a framework that couples generative human motion with an explicit physical object simulation.<n>Experiments show PhyGenHOI generates physically consistent 4D HOI across diverse actions, humans, and objects, outperforming baselines.
  arXiv  Detail & Related papers  (2026-05-28T17:29:19Z)
• UniCon3R: Contact-aware 3D Human-Scene Reconstruction from Monocular Video [82.5562736830041]
  We introduce UniCon3R, a unified feed-forward framework for online human-scene 4D reconstruction from monocular videos.<n>We explicitly model interaction by inferring 3D contact from the human pose and scene geometry.<n>This enables UniCon3R to jointly recover high-fidelity scene geometry and spatially aligned 3D humans within the scene.
  arXiv  Detail & Related papers  (2026-04-21T19:06:50Z)
• ArtHOI: Articulated Human-Object Interaction Synthesis by 4D Reconstruction from Video Priors [51.06020148149403]
  We introduce ArtHOI, the first zero-shot framework for articulated human-object interaction synthesis via 4D reconstruction from video priors.<n>ArtHOI bridges video-based generation and geometry-aware reconstruction, producing interactions that are both semantically aligned and physically grounded.
  arXiv  Detail & Related papers  (2026-03-04T17:58:04Z)
• MeshMimic: Geometry-Aware Humanoid Motion Learning through 3D Scene Reconstruction [54.36564144414704]
  MeshMimic is an innovative framework that bridges 3D scene reconstruction and embodied intelligence to enable humanoid robots to learn coupled "motion-terrain" interactions directly from video.<n>By leveraging state-of-the-art 3D vision models, our framework precisely segments and reconstructs both human trajectories and the underlying 3D geometry of terrains and objects.
  arXiv  Detail & Related papers  (2026-02-17T17:09:45Z)
• CRISP: Contact-Guided Real2Sim from Monocular Video with Planar Scene Primitives [65.89192712575797]
  We introduce CRISP, a method that recovers simulatable human motion and scene geometry from monocular video.<n>Our approach reduces motion tracking failure rates from 55.2% to 6.9% on human-centric video benchmarks.<n>This demonstrates CRISP's ability to generate physically-valid human motion and interaction environments at scale.
  arXiv  Detail & Related papers  (2025-12-16T18:59:50Z)
• Gravity-Aware Monocular 3D Human-Object Reconstruction [73.25185274561139]
  This paper proposes a new approach for joint markerless 3D human motion capture and object trajectory estimation from monocular RGB videos. We focus on scenes with objects partially observed during a free flight. In the experiments, our approach achieves state-of-the-art accuracy in 3D human motion capture on various metrics.
  arXiv  Detail & Related papers  (2021-08-19T17:59:57Z)
• Kinematics-Guided Reinforcement Learning for Object-Aware 3D Ego-Pose Estimation [25.03715978502528]
  We propose a method for incorporating object interaction and human body dynamics into the task of 3D ego-pose estimation. We use a kinematics model of the human body to represent the entire range of human motion, and a dynamics model of the body to interact with objects inside a physics simulator. This is the first work to estimate a physically valid 3D full-body interaction sequence with objects from egocentric videos.
  arXiv  Detail & Related papers  (2020-11-10T00:06:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.