InterTrack: Tracking Human Object Interaction without Object Templates

• URL: http://arxiv.org/abs/2408.13953v1
• Date: Sun, 25 Aug 2024 22:26:46 GMT
• Title: InterTrack: Tracking Human Object Interaction without Object Templates
• Authors: Xianghui Xie, Jan Eric Lenssen, Gerard Pons-Moll,
• Abstract summary: We present a method to track human object interaction without any object shape templates. We decompose the 4D tracking problem into per-frame pose tracking and canonical shape optimization. Our method significantly outperforms previous template-based video tracking and single-frame reconstruction methods.
• Score: 34.31283776812698
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Tracking human object interaction from videos is important to understand human behavior from the rapidly growing stream of video data. Previous video-based methods require predefined object templates while single-image-based methods are template-free but lack temporal consistency. In this paper, we present a method to track human object interaction without any object shape templates. We decompose the 4D tracking problem into per-frame pose tracking and canonical shape optimization. We first apply a single-view reconstruction method to obtain temporally-inconsistent per-frame interaction reconstructions. Then, for the human, we propose an efficient autoencoder to predict SMPL vertices directly from the per-frame reconstructions, introducing temporally consistent correspondence. For the object, we introduce a pose estimator that leverages temporal information to predict smooth object rotations under occlusions. To train our model, we propose a method to generate synthetic interaction videos and synthesize in total 10 hour videos of 8.5k sequences with full 3D ground truth. Experiments on BEHAVE and InterCap show that our method significantly outperforms previous template-based video tracking and single-frame reconstruction methods. Our proposed synthetic video dataset also allows training video-based methods that generalize to real-world videos. Our code and dataset will be publicly released.

Related papers

• DressRecon: Freeform 4D Human Reconstruction from Monocular Video [64.61230035671885]
  We present a method to reconstruct time-consistent human body models from monocular videos. We focus on extremely loose clothing or handheld object interactions. DressRecon yields higher-fidelity 3D reconstructions than prior art.
  arXiv  Detail & Related papers  (2024-09-30T17:59:15Z)
• Diffusion-Guided Reconstruction of Everyday Hand-Object Interaction Clips [38.02945794078731]
  We tackle the task of reconstructing hand-object interactions from short video clips. Our approach casts 3D inference as a per-video optimization and recovers a neural 3D representation of the object shape. We empirically evaluate our approach on egocentric videos, and observe significant improvements over prior single-view and multi-view methods.
  arXiv  Detail & Related papers  (2023-09-11T17:58:30Z)
• Visibility Aware Human-Object Interaction Tracking from Single RGB Camera [40.817960406002506]
  We propose a novel method to track the 3D human, object, contacts between them, and their relative translation across frames from a single RGB camera. We condition our neural field reconstructions for human and object on per-frame SMPL model estimates obtained by pre-fitting SMPL to a video sequence. Human and object motion from visible frames provides valuable information to infer the occluded object.
  arXiv  Detail & Related papers  (2023-03-29T06:23:44Z)
• BundleSDF: Neural 6-DoF Tracking and 3D Reconstruction of Unknown Objects [89.2314092102403]
  We present a near real-time method for 6-DoF tracking of an unknown object from a monocular RGBD video sequence. Our method works for arbitrary rigid objects, even when visual texture is largely absent.
  arXiv  Detail & Related papers  (2023-03-24T17:13:49Z)
• Learning Dynamic View Synthesis With Few RGBD Cameras [60.36357774688289]
  We propose to utilize RGBD cameras to synthesize free-viewpoint videos of dynamic indoor scenes. We generate point clouds from RGBD frames and then render them into free-viewpoint videos via a neural feature. We introduce a simple Regional Depth-Inpainting module that adaptively inpaints missing depth values to render complete novel views.
  arXiv  Detail & Related papers  (2022-04-22T03:17:35Z)
• Estimating 3D Motion and Forces of Human-Object Interactions from Internet Videos [49.52070710518688]
  We introduce a method to reconstruct the 3D motion of a person interacting with an object from a single RGB video. Our method estimates the 3D poses of the person together with the object pose, the contact positions and the contact forces on the human body.
  arXiv  Detail & Related papers  (2021-11-02T13:40:18Z)
• LASR: Learning Articulated Shape Reconstruction from a Monocular Video [97.92849567637819]
  We introduce a template-free approach to learn 3D shapes from a single video. Our method faithfully reconstructs nonrigid 3D structures from videos of human, animals, and objects of unknown classes.
  arXiv  Detail & Related papers  (2021-05-06T21:41:11Z)
• Self-Attentive 3D Human Pose and Shape Estimation from Videos [82.63503361008607]
  We present a video-based learning algorithm for 3D human pose and shape estimation. We exploit temporal information in videos and propose a self-attention module. We evaluate our method on the 3DPW, MPI-INF-3DHP, and Human3.6M datasets.
  arXiv  Detail & Related papers  (2021-03-26T00:02:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.