WHAC: World-grounded Humans and Cameras

• URL: http://arxiv.org/abs/2403.12959v1
• Date: Tue, 19 Mar 2024 17:58:02 GMT
• Title: WHAC: World-grounded Humans and Cameras
• Authors: Wanqi Yin, Zhongang Cai, Ruisi Wang, Fanzhou Wang, Chen Wei, Haiyi Mei, Weiye Xiao, Zhitao Yang, Qingping Sun, Atsushi Yamashita, Ziwei Liu, Lei Yang,
• Abstract summary: We aim to recover expressive parametric human models (i.e., SMPL-X) and corresponding camera poses jointly. We introduce a novel framework, referred to as WHAC, to facilitate world-grounded expressive human pose and shape estimation. We present a new synthetic dataset, WHAC-A-Mole, which includes accurately annotated humans and cameras.
• Score: 37.877565981937586
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Estimating human and camera trajectories with accurate scale in the world coordinate system from a monocular video is a highly desirable yet challenging and ill-posed problem. In this study, we aim to recover expressive parametric human models (i.e., SMPL-X) and corresponding camera poses jointly, by leveraging the synergy between three critical players: the world, the human, and the camera. Our approach is founded on two key observations. Firstly, camera-frame SMPL-X estimation methods readily recover absolute human depth. Secondly, human motions inherently provide absolute spatial cues. By integrating these insights, we introduce a novel framework, referred to as WHAC, to facilitate world-grounded expressive human pose and shape estimation (EHPS) alongside camera pose estimation, without relying on traditional optimization techniques. Additionally, we present a new synthetic dataset, WHAC-A-Mole, which includes accurately annotated humans and cameras, and features diverse interactive human motions as well as realistic camera trajectories. Extensive experiments on both standard and newly established benchmarks highlight the superiority and efficacy of our framework. We will make the code and dataset publicly available.

Related papers

• Simultaneously Recovering Multi-Person Meshes and Multi-View Cameras with Human Semantics [14.385538705947782]
  We focus on multi-person motion capture with uncalibrated cameras. Key idea is to incorporate motion prior knowledge to simultaneously estimate camera parameters and human meshes. Results show that accurate camera parameters and human motions can be obtained through a one-step reconstruction.
  arXiv  Detail & Related papers  (2024-12-25T05:35:30Z)
• Reconstructing People, Places, and Cameras [57.81696692335401]
  "Humans and Structure from Motion" (HSfM) is a method for jointly reconstructing multiple human meshes, scene point clouds, and camera parameters in a metric world coordinate system. Our results show that incorporating human data into the SfM pipeline improves camera pose estimation.
  arXiv  Detail & Related papers  (2024-12-23T18:58:34Z)
• PACE: Human and Camera Motion Estimation from in-the-wild Videos [113.76041632912577]
  We present a method to estimate human motion in a global scene from moving cameras. This is a highly challenging task due to the coupling of human and camera motions in the video. We propose a joint optimization framework that disentangles human and camera motions using both foreground human motion priors and background scene features.
  arXiv  Detail & Related papers  (2023-10-20T19:04:14Z)
• Scene-Aware 3D Multi-Human Motion Capture from a Single Camera [83.06768487435818]
  We consider the problem of estimating the 3D position of multiple humans in a scene as well as their body shape and articulation from a single RGB video recorded with a static camera. We leverage recent advances in computer vision using large-scale pre-trained models for a variety of modalities, including 2D body joints, joint angles, normalized disparity maps, and human segmentation masks. In particular, we estimate the scene depth and unique person scale from normalized disparity predictions using the 2D body joints and joint angles.
  arXiv  Detail & Related papers  (2023-01-12T18:01:28Z)
• Embodied Scene-aware Human Pose Estimation [25.094152307452]
  We propose embodied scene-aware human pose estimation. Our method is one stage, causal, and recovers global 3D human poses in a simulated environment.
  arXiv  Detail & Related papers  (2022-06-18T03:50:19Z)
• GLAMR: Global Occlusion-Aware Human Mesh Recovery with Dynamic Cameras [99.07219478953982]
  We present an approach for 3D global human mesh recovery from monocular videos recorded with dynamic cameras. We first propose a deep generative motion infiller, which autoregressively infills the body motions of occluded humans based on visible motions. In contrast to prior work, our approach reconstructs human meshes in consistent global coordinates even with dynamic cameras.
  arXiv  Detail & Related papers  (2021-12-02T18:59:54Z)
• Camera Motion Agnostic 3D Human Pose Estimation [8.090223360924004]
  This paper presents a camera motion agnostic approach for predicting 3D human pose and mesh defined in the world coordinate system. We propose a network based on bidirectional gated recurrent units (GRUs) that predicts the global motion sequence from the local pose sequence. We use 3DPW and synthetic datasets, which are constructed in a moving-camera environment, for evaluation.
  arXiv  Detail & Related papers  (2021-12-01T08:22:50Z)
• TRiPOD: Human Trajectory and Pose Dynamics Forecasting in the Wild [77.59069361196404]
  TRiPOD is a novel method for predicting body dynamics based on graph attentional networks. To incorporate a real-world challenge, we learn an indicator representing whether an estimated body joint is visible/invisible at each frame. Our evaluation shows that TRiPOD outperforms all prior work and state-of-the-art specifically designed for each of the trajectory and pose forecasting tasks.
  arXiv  Detail & Related papers  (2021-04-08T20:01:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.