MUC: Mixture of Uncalibrated Cameras for Robust 3D Human Body Reconstruction

• URL: http://arxiv.org/abs/2403.05055v3
• Date: Sat, 24 Aug 2024 06:40:17 GMT
• Title: MUC: Mixture of Uncalibrated Cameras for Robust 3D Human Body Reconstruction
• Authors: Yitao Zhu, Sheng Wang, Mengjie Xu, Zixu Zhuang, Zhixin Wang, Kaidong Wang, Han Zhang, Qian Wang,
• Abstract summary: Multiple cameras can provide comprehensive multi-view video coverage of a person. Previous studies have overlooked the challenges posed by self-occlusion under multiple views. We introduce a method to reconstruct the 3D human body from multiple uncalibrated camera views.
• Score: 12.942635715952525
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multiple cameras can provide comprehensive multi-view video coverage of a person. Fusing this multi-view data is crucial for tasks like behavioral analysis, although it traditionally requires camera calibration, a process that is often complex. Moreover, previous studies have overlooked the challenges posed by self-occlusion under multiple views and the continuity of human body shape estimation. In this study, we introduce a method to reconstruct the 3D human body from multiple uncalibrated camera views. Initially, we utilize a pre-trained human body encoder to process each camera view individually, enabling the reconstruction of human body models and parameters for each view along with predicted camera positions. Rather than merely averaging the models across views, we develop a neural network trained to assign weights to individual views for all human body joints, based on the estimated distribution of joint distances from each camera. Additionally, we focus on the mesh surface of the human body for dynamic fusion, allowing for the seamless integration of facial expressions and body shape into a unified human body model. Our method has shown excellent performance in reconstructing the human body on two public datasets, advancing beyond previous work from the SMPL model to the SMPL-X model. This extension incorporates more complex hand poses and facial expressions, enhancing the detail and accuracy of the reconstructions. Crucially, it supports the flexible ad-hoc deployment of any number of cameras, offering significant potential for various applications. Our code is available at https://github.com/AbsterZhu/MUC.

Related papers

• DressRecon: Freeform 4D Human Reconstruction from Monocular Video [64.61230035671885]
  We present a method to reconstruct time-consistent human body models from monocular videos. We focus on extremely loose clothing or handheld object interactions. DressRecon yields higher-fidelity 3D reconstructions than prior art.
  arXiv  Detail & Related papers  (2024-09-30T17:59:15Z)
• Self-learning Canonical Space for Multi-view 3D Human Pose Estimation [57.969696744428475]
  Multi-view 3D human pose estimation is naturally superior to single view one. The accurate annotation of these information is hard to obtain. We propose a fully self-supervised framework, named cascaded multi-view aggregating network (CMANet) CMANet is superior to state-of-the-art methods in extensive quantitative and qualitative analysis.
  arXiv  Detail & Related papers  (2024-03-19T04:54:59Z)
• Human Mesh Recovery from Arbitrary Multi-view Images [57.969696744428475]
  We propose a divide and conquer framework for Unified Human Mesh Recovery (U-HMR) from arbitrary multi-view images. In particular, U-HMR consists of a decoupled structure and two main components: camera and body decoupling (CBD), camera pose estimation (CPE) and arbitrary view fusion (AVF) We conduct extensive experiments on three public datasets: Human3.6M, MPI-INF-3DHP, and TotalCapture.
  arXiv  Detail & Related papers  (2024-03-19T04:47:56Z)
• Multi-HMR: Multi-Person Whole-Body Human Mesh Recovery in a Single Shot [22.848563931757962]
  We present Multi-HMR, a strong sigle-shot model for multi-person 3D human mesh recovery from a single RGB image. Predictions encompass the whole body, including hands and facial expressions, using the SMPL-X parametric model. We show that incorporating it into the training data further enhances predictions, particularly for hands.
  arXiv  Detail & Related papers  (2024-02-22T16:05:13Z)
• Probabilistic Human Mesh Recovery in 3D Scenes from Egocentric Views [32.940614931864154]
  We propose a scene-conditioned diffusion method to model the body pose distribution. The method generates bodies in plausible human-scene interactions. It achieves superior accuracy for visible joints and diversity for invisible body parts.
  arXiv  Detail & Related papers  (2023-04-12T17:58:57Z)
• Scene-Aware 3D Multi-Human Motion Capture from a Single Camera [83.06768487435818]
  We consider the problem of estimating the 3D position of multiple humans in a scene as well as their body shape and articulation from a single RGB video recorded with a static camera. We leverage recent advances in computer vision using large-scale pre-trained models for a variety of modalities, including 2D body joints, joint angles, normalized disparity maps, and human segmentation masks. In particular, we estimate the scene depth and unique person scale from normalized disparity predictions using the 2D body joints and joint angles.
  arXiv  Detail & Related papers  (2023-01-12T18:01:28Z)
• Human Performance Capture from Monocular Video in the Wild [50.34917313325813]
  We propose a method capable of capturing the dynamic 3D human shape from a monocular video featuring challenging body poses. Our method outperforms state-of-the-art methods on an in-the-wild human video dataset 3DPW.
  arXiv  Detail & Related papers  (2021-11-29T16:32:41Z)
• Self-Supervised 3D Human Pose Estimation with Multiple-View Geometry [2.7541825072548805]
  We present a self-supervised learning algorithm for 3D human pose estimation of a single person based on a multiple-view camera system. We propose a four-loss function learning algorithm, which does not require any 2D or 3D body pose ground-truth.
  arXiv  Detail & Related papers  (2021-08-17T17:31:24Z)
• Self-Supervised 3D Human Pose Estimation via Part Guided Novel Image Synthesis [72.34794624243281]
  We propose a self-supervised learning framework to disentangle variations from unlabeled video frames. Our differentiable formalization, bridging the representation gap between the 3D pose and spatial part maps, allows us to operate on videos with diverse camera movements.
  arXiv  Detail & Related papers  (2020-04-09T07:55:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.