HAT-4D: Lifting Monocular Video for 4D Multi-Object Interactions via Human-Agent Collaboration

• URL: http://arxiv.org/abs/2606.28215v1
• Date: Fri, 26 Jun 2026 16:05:58 GMT
• Title: HAT-4D: Lifting Monocular Video for 4D Multi-Object Interactions via Human-Agent Collaboration
• Abstract summary: HAT-4D reconstructs the 3D geometry, temporal dynamics, and physical interactions of multiple objects from a single video.<n>As a scalable data engine, HAT-4D facilitates the creation of MVOIK-4D, an open-world benchmark for monocular 4D interaction reconstruction.
• Score: 38.16996826492207
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Extracting dynamic 4D object interactions from massive, in-the-wild monocular videos offers a highly efficient data collection pathway for scaling Embodied AI and training VLAs. However, existing monocular 4D reconstruction methods primarily focus on isolated objects, often failing under the severe occlusions and complex dynamics inherent in multi-object interactions. To bridge this gap, we propose HAT-4D, the first agentic framework designed to reconstruct the 3D geometry, temporal dynamics, and physical interactions of multiple objects from a single video. By integrating VLMs with a multi-level human-in-the-loop feedback mechanism, HAT-4D efficiently resolves depth ambiguities and interaction-induced occlusions during 3D generation and 4D propagation, yielding physically plausible assets without relying on expensive multicamera rigs. As a scalable data engine, HAT-4D facilitates the creation of MVOIK-4D, an open-world benchmark for monocular 4D interaction reconstruction, accompanied by a novel multi-dimensional evaluation protocol focused on physical plausibility and temporal consistency. Extensive experiments demonstrate that HAT-4D achieves SOTA performance on most evaluation metrics, while maintaining competitive semantic alignment. Ablation studies show that introducing a small amount of human feedback improves interaction reconstruction. Moreover, the data produced by HAT-4D effectively improves baseline performance when used for fine-tuning. Our data and code are available at https://lijiaxin0111.github.io/HAT4D/

Related papers

• High-Fidelity 4D Hand-Object Capture via Multi-View Spatiotemporal Tracking and Physics-Aware Gaussians [38.0162622592412]
  We propose a novel system for the robust and accurate reconstruction of hands and objects from synchronized and calibrated multi-view videos.<n>Our pipeline achieves highly robust, artifact-free reconstruction, providing an efficient foundation for automated 4D asset generation.
  arXiv  Detail & Related papers  (2026-06-14T16:36:49Z)
• Full-4D: Generating Full-Scope 4D Scenes from a Single-View Video [63.61099683012546]
  We introduce RealMV-4D, a dataset of synchronized multi-view videos.<n>We then train a multi-view video diffusion model driven by a novel fused time(T)-view(V) attention mechanism.<n>Experiments demonstrate that our method outperforms existing approaches in both visual fidelity and consistency.
  arXiv  Detail & Related papers  (2026-05-25T06:59:26Z)
• Syn4D: A Multiview Synthetic 4D Dataset [92.50132053229817]
  3D reconstruction and tracking of dynamic scenes from monocular video remains an important open challenge in computer vision.<n>To address this limitation, we introduce Syn4D, a multiview synthetic dynamic scenes.<n>Key feature of Syn4D is the ability to unproject any pixel into 3D.
  arXiv  Detail & Related papers  (2026-05-06T17:59:55Z)
• MLLM-4D: Towards Visual-based Spatial-Temporal Intelligence [50.11889361459544]
  Humans are born with vision-based 4D spatial-temporal intelligence.<n>Despite its importance, this capability remains a significant bottleneck for current large language models (MLLMs)
  arXiv  Detail & Related papers  (2026-02-28T07:23:36Z)
• Motion 3-to-4: 3D Motion Reconstruction for 4D Synthesis [53.48281548500864]
  Motion 3-to-4 is a feed-forward framework for synthesising high-quality 4D dynamic objects from a single monocular video.<n>Our model learns a compact motion latent representation and predicts per-frame trajectories to recover complete robustness, temporally coherent geometry.
  arXiv  Detail & Related papers  (2026-01-20T18:59:48Z)
• SWiT-4D: Sliding-Window Transformer for Lossless and Parameter-Free Temporal 4D Generation [30.72482055095692]
  SWiT-4D is a Sliding-Window Transformer for lossless, parameter-free temporal 4D mesh generation.<n> SWiT-4D integrates seamlessly with any Diffusion Transformer (DiT)-based image-to-3D generator.<n>It achieves high-fidelity geometry and stable temporal consistency, indicating practical deployability under extremely limited 4D supervision.
  arXiv  Detail & Related papers  (2025-12-11T17:54:31Z)
• Efficient and Scalable Monocular Human-Object Interaction Motion Reconstruction [19.16200327159635]
  Generalized robots must learn from diverse, large-scale humanobject interactions (HOI) to robustly operate in the real world.<n>We introduce 4DHOISolver, a novel and efficient optimization framework that constrains ill-posed 4D HOI reconstruction problem.<n>We introduce Open4DHOI, a new large-scale 4D HOI dataset featuring a diverse catalog of 144 object types and 103 actions.
  arXiv  Detail & Related papers  (2025-11-30T16:21:47Z)
• GenHOI: Generalizing Text-driven 4D Human-Object Interaction Synthesis for Unseen Objects [13.830968058014546]
  GenHOI is a two-stage framework aimed at achieving two key objectives: 1) generalization to unseen objects and 2) the synthesis of high-fidelity 4D HOI sequences.<n>We introduce a Contact-Aware Diffusion Model (ContactDM) in the second stage to seamlessly interpolate 3D HOIs into densely temporally coherent 4D HOI sequences.<n> Experimental results show that we achieve state-of-the-art results on the publicly available OMOMODM and 3D-FUTURE datasets.
  arXiv  Detail & Related papers  (2025-06-18T14:17:53Z)
• Diffusion4D: Fast Spatial-temporal Consistent 4D Generation via Video Diffusion Models [116.31344506738816]
  We present a novel framework, textbfDiffusion4D, for efficient and scalable 4D content generation. We develop a 4D-aware video diffusion model capable of synthesizing orbital views of dynamic 3D assets. Our method surpasses prior state-of-the-art techniques in terms of generation efficiency and 4D geometry consistency.
  arXiv  Detail & Related papers  (2024-05-26T17:47:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.