End-to-End Spatial-Temporal Transformer for Real-time 4D HOI Reconstruction

• URL: http://arxiv.org/abs/2603.14435v1
• Date: Sun, 15 Mar 2026 15:21:36 GMT
• Title: End-to-End Spatial-Temporal Transformer for Real-time 4D HOI Reconstruction
• Abstract summary: THO is an end-to-end Spatial-Temporal Transformer that predicts human motion and coordinated object motion in a forward fashion from the given video and 3D template.<n> experiments demonstrate that THO operates at an inference speed of 31.5 FPS on a single 4090 GPU, achieving a >600x speedup over prior optimization-based methods.
• Score: 74.31251139839047
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Monocular 4D human-object interaction (HOI) reconstruction - recovering a moving human and a manipulated object from a single RGB video - remains challenging due to depth ambiguity and frequent occlusions. Existing methods often rely on multi-stage pipelines or iterative optimization, leading to high inference latency, failing to meet real-time requirements, and susceptibility to error accumulation. To address these limitations, we propose THO, an end-to-end Spatial-Temporal Transformer that predicts human motion and coordinated object motion in a forward fashion from the given video and 3D template. THO achieves this by leveraging spatial-temporal HOI tuple priors. Spatial priors exploit contact-region proximity to infer occluded object features from human cues, while temporal priors capture cross-frame kinematic correlations to refine object representations and enforce physical coherence. Extensive experiments demonstrate that THO operates at an inference speed of 31.5 FPS on a single RTX 4090 GPU, achieving a >600x speedup over prior optimization-based methods while simultaneously improving reconstruction accuracy and temporal consistency. The project page is available at: https://nianheng.github.io/THO-project/

Related papers

• HandFlow: Fully Generative 4D Hand Recovery with Flow Matching [25.56265773611543]
  HandFlow is a fully generative flow-matching framework for temporally coherent 3D hand pose and shape estimation from monocular video.<n>It achieves state-of-the-art performance, with particularly large gains in world-space accuracy and temporal smoothness.<n>On a single GPU HandFlow reconstructs a 150-frame sequence at 47 fps, about 12x faster than the fastest prior video-based method.
  arXiv  Detail & Related papers  (2026-07-13T08:13:42Z)
• SLAT-Phys: Fast Material Property Field Prediction from Structured 3D Latents [55.26273954022457]
  Estimating the material property field of 3D assets is critical for physics-based simulation, robotics, and digital twin generation.<n>We present SLAT-Phys, an end-to-end method that predicts spatially varying material property fields of 3D assets directly from a single RGB image without explicit 3D reconstruction.
  arXiv  Detail & Related papers  (2026-03-25T06:14:03Z)
• GeoMotion: Rethinking Motion Segmentation via Latent 4D Geometry [61.24189040578178]
  We propose a fully learning-based approach that directly infers moving objects from latent feature representations via attention mechanisms.<n>Our key insight is to bypass explicit correspondence estimation and instead let the model learn to implicitly disentangle object and camera motion.<n>Our approach achieves state-of-the-art motion segmentation performance with high efficiency.
  arXiv  Detail & Related papers  (2026-02-25T11:36:33Z)
• CARI4D: Category Agnostic 4D Reconstruction of Human-Object Interaction [40.557276644446475]
  We present CARI4D, the first category-agnostic method that reconstructs spatially and temporarily consistent 4D human-object interaction at metric scale from monocular RGB videos.<n>Our model generalizes beyond the training categories and thus can be applied zero-shot to in-the-wild internet videos.
  arXiv  Detail & Related papers  (2025-12-12T19:11:11Z)
• Forge4D: Feed-Forward 4D Human Reconstruction and Interpolation from Uncalibrated Sparse-view Videos [27.595035122927204]
  We present a feed-forward 4D human reconstruction and model that efficientlycalibrates temporally aligned representations from uncalibrated sparse-view videos.<n>For novel time, we design a novel motion prediction module to predict dense motions for each 3D Gaussian between two adjacent frames.<n>Experiments demonstrate the effectiveness of our model on both in-domain and out-of-domain datasets.
  arXiv  Detail & Related papers  (2025-09-29T02:47:14Z)
• InterTrack: Tracking Human Object Interaction without Object Templates [34.31283776812698]
  We present a method to track human object interaction without any object shape templates. We decompose the 4D tracking problem into per-frame pose tracking and canonical shape optimization. Our method significantly outperforms previous template-based video tracking and single-frame reconstruction methods.
  arXiv  Detail & Related papers  (2024-08-25T22:26:46Z)
• Consistent4D: Consistent 360{\deg} Dynamic Object Generation from Monocular Video [15.621374353364468]
  Consistent4D is a novel approach for generating 4D dynamic objects from uncalibrated monocular videos. We cast the 360-degree dynamic object reconstruction as a 4D generation problem, eliminating the need for tedious multi-view data collection and camera calibration.
  arXiv  Detail & Related papers  (2023-11-06T03:26:43Z)
• Coordinate Transformer: Achieving Single-stage Multi-person Mesh Recovery from Videos [91.44553585470688]
  Multi-person 3D mesh recovery from videos is a critical first step towards automatic perception of group behavior in virtual reality, physical therapy and beyond. We propose the Coordinate transFormer (CoordFormer) that directly models multi-person spatial-temporal relations and simultaneously performs multi-mesh recovery in an end-to-end manner. Experiments on the 3DPW dataset demonstrate that CoordFormer significantly improves the state-of-the-art, outperforming the previously best results by 4.2%, 8.8% and 4.7% according to the MPJPE, PAMPJPE, and PVE metrics, respectively.
  arXiv  Detail & Related papers  (2023-08-20T18:23:07Z)
• Learning Dynamic View Synthesis With Few RGBD Cameras [60.36357774688289]
  We propose to utilize RGBD cameras to synthesize free-viewpoint videos of dynamic indoor scenes. We generate point clouds from RGBD frames and then render them into free-viewpoint videos via a neural feature. We introduce a simple Regional Depth-Inpainting module that adaptively inpaints missing depth values to render complete novel views.
  arXiv  Detail & Related papers  (2022-04-22T03:17:35Z)
• OcclusionFusion: Occlusion-aware Motion Estimation for Real-time Dynamic 3D Reconstruction [14.130915525776055]
  RGBD-based real-time dynamic 3D reconstruction suffers from inaccurate inter-frame motion estimation. We propose OcclusionFusion, a novel method to calculate occlusion-aware 3D motion to guide the reconstruction. Our technique outperforms existing single-view-based real-time methods by a large margin.
  arXiv  Detail & Related papers  (2022-03-15T15:09:01Z)
• Motion-from-Blur: 3D Shape and Motion Estimation of Motion-blurred Objects in Videos [115.71874459429381]
  We propose a method for jointly estimating the 3D motion, 3D shape, and appearance of highly motion-blurred objects from a video. Experiments on benchmark datasets demonstrate that our method outperforms previous methods for fast moving object deblurring and 3D reconstruction.
  arXiv  Detail & Related papers  (2021-11-29T11:25:14Z)
• Spatiotemporal Bundle Adjustment for Dynamic 3D Human Reconstruction in the Wild [49.672487902268706]
  We present a framework that jointly estimates camera temporal alignment and 3D point triangulation. We reconstruct 3D motion trajectories of human bodies in events captured by multiple unsynchronized and unsynchronized video cameras.
  arXiv  Detail & Related papers  (2020-07-24T23:50:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.