4D Human Body Capture from Egocentric Video via 3D Scene Grounding

• URL: http://arxiv.org/abs/2011.13341v2
• Date: Fri, 15 Oct 2021 23:03:13 GMT
• Title: 4D Human Body Capture from Egocentric Video via 3D Scene Grounding
• Authors: Miao Liu, Dexin Yang, Yan Zhang, Zhaopeng Cui, James M. Rehg, Siyu Tang
• Abstract summary: We introduce a novel task of reconstructing a time series of second-person 3D human body meshes from monocular egocentric videos. The unique viewpoint and rapid embodied camera motion of egocentric videos raise additional technical barriers for human body capture.
• Score: 38.3169520384642
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We introduce a novel task of reconstructing a time series of second-person 3D human body meshes from monocular egocentric videos. The unique viewpoint and rapid embodied camera motion of egocentric videos raise additional technical barriers for human body capture. To address those challenges, we propose a simple yet effective optimization-based approach that leverages 2D observations of the entire video sequence and human-scene interaction constraint to estimate second-person human poses, shapes, and global motion that are grounded on the 3D environment captured from the egocentric view. We conduct detailed ablation studies to validate our design choice. Moreover, we compare our method with the previous state-of-the-art method on human motion capture from monocular video, and show that our method estimates more accurate human-body poses and shapes under the challenging egocentric setting. In addition, we demonstrate that our approach produces more realistic human-scene interaction.

Related papers

• AMG: Avatar Motion Guided Video Generation [5.82136706118236]
  We propose AMG, a method that combines the 2D photorealism and 3D controllability by conditioning video diffusion models on controlled rendering of 3D avatars. AMG is the first method that enables multi-person diffusion video generation with precise control over camera positions, human motions, and background style.
  arXiv  Detail & Related papers  (2024-09-02T23:59:01Z)
• MultiPly: Reconstruction of Multiple People from Monocular Video in the Wild [32.6521941706907]
  We present MultiPly, a novel framework to reconstruct multiple people in 3D from monocular in-the-wild videos. We first define a layered neural representation for the entire scene, composited by individual human and background models. We learn the layered neural representation from videos via our layer-wise differentiable volume rendering.
  arXiv  Detail & Related papers  (2024-06-03T17:59:57Z)
• 3D Human Pose Perception from Egocentric Stereo Videos [67.9563319914377]
  We propose a new transformer-based framework to improve egocentric stereo 3D human pose estimation. Our method is able to accurately estimate human poses even in challenging scenarios, such as crouching and sitting. We will release UnrealEgo2, UnrealEgo-RW, and trained models on our project page.
  arXiv  Detail & Related papers  (2023-12-30T21:21:54Z)
• Scene-Aware 3D Multi-Human Motion Capture from a Single Camera [83.06768487435818]
  We consider the problem of estimating the 3D position of multiple humans in a scene as well as their body shape and articulation from a single RGB video recorded with a static camera. We leverage recent advances in computer vision using large-scale pre-trained models for a variety of modalities, including 2D body joints, joint angles, normalized disparity maps, and human segmentation masks. In particular, we estimate the scene depth and unique person scale from normalized disparity predictions using the 2D body joints and joint angles.
  arXiv  Detail & Related papers  (2023-01-12T18:01:28Z)
• Ego-Body Pose Estimation via Ego-Head Pose Estimation [22.08240141115053]
  Estimating 3D human motion from an egocentric video sequence plays a critical role in human behavior understanding and has various applications in VR/AR. We propose a new method, Ego-Body Pose Estimation via Ego-Head Pose Estimation (EgoEgo), which decomposes the problem into two stages, connected by the head motion as an intermediate representation. This disentanglement of head and body pose eliminates the need for training datasets with paired egocentric videos and 3D human motion.
  arXiv  Detail & Related papers  (2022-12-09T02:25:20Z)
• Human Performance Capture from Monocular Video in the Wild [50.34917313325813]
  We propose a method capable of capturing the dynamic 3D human shape from a monocular video featuring challenging body poses. Our method outperforms state-of-the-art methods on an in-the-wild human video dataset 3DPW.
  arXiv  Detail & Related papers  (2021-11-29T16:32:41Z)
• Action2video: Generating Videos of Human 3D Actions [31.665831044217363]
  We aim to tackle the interesting yet challenging problem of generating videos of diverse and natural human motions from prescribed action categories. Key issue lies in the ability to synthesize multiple distinct motion sequences that are realistic in their visual appearances. Action2motionally generates plausible 3D pose sequences of a prescribed action category, which are processed and rendered by motion2video to form 2D videos.
  arXiv  Detail & Related papers  (2021-11-12T20:20:37Z)
• Estimating 3D Motion and Forces of Human-Object Interactions from Internet Videos [49.52070710518688]
  We introduce a method to reconstruct the 3D motion of a person interacting with an object from a single RGB video. Our method estimates the 3D poses of the person together with the object pose, the contact positions and the contact forces on the human body.
  arXiv  Detail & Related papers  (2021-11-02T13:40:18Z)
• Contact and Human Dynamics from Monocular Video [73.47466545178396]
  Existing deep models predict 2D and 3D kinematic poses from video that are approximately accurate, but contain visible errors. We present a physics-based method for inferring 3D human motion from video sequences that takes initial 2D and 3D pose estimates as input.
  arXiv  Detail & Related papers  (2020-07-22T21:09:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.