Multi-HMR: Multi-Person Whole-Body Human Mesh Recovery in a Single Shot

• URL: http://arxiv.org/abs/2402.14654v2
• Date: Wed, 24 Jul 2024 09:55:25 GMT
• Title: Multi-HMR: Multi-Person Whole-Body Human Mesh Recovery in a Single Shot
• Authors: Fabien Baradel, Matthieu Armando, Salma Galaaoui, Romain Brégier, Philippe Weinzaepfel, Grégory Rogez, Thomas Lucas,
• Abstract summary: We present Multi-HMR, a strong sigle-shot model for multi-person 3D human mesh recovery from a single RGB image. Predictions encompass the whole body, including hands and facial expressions, using the SMPL-X parametric model. We show that incorporating it into the training data further enhances predictions, particularly for hands.
• Score: 22.848563931757962
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present Multi-HMR, a strong sigle-shot model for multi-person 3D human mesh recovery from a single RGB image. Predictions encompass the whole body, i.e., including hands and facial expressions, using the SMPL-X parametric model and 3D location in the camera coordinate system. Our model detects people by predicting coarse 2D heatmaps of person locations, using features produced by a standard Vision Transformer (ViT) backbone. It then predicts their whole-body pose, shape and 3D location using a new cross-attention module called the Human Prediction Head (HPH), with one query attending to the entire set of features for each detected person. As direct prediction of fine-grained hands and facial poses in a single shot, i.e., without relying on explicit crops around body parts, is hard to learn from existing data, we introduce CUFFS, the Close-Up Frames of Full-Body Subjects dataset, containing humans close to the camera with diverse hand poses. We show that incorporating it into the training data further enhances predictions, particularly for hands. Multi-HMR also optionally accounts for camera intrinsics, if available, by encoding camera ray directions for each image token. This simple design achieves strong performance on whole-body and body-only benchmarks simultaneously: a ViT-S backbone on $448{\times}448$ images already yields a fast and competitive model, while larger models and higher resolutions obtain state-of-the-art results.

Related papers

• Self-learning Canonical Space for Multi-view 3D Human Pose Estimation [57.969696744428475]
  Multi-view 3D human pose estimation is naturally superior to single view one. The accurate annotation of these information is hard to obtain. We propose a fully self-supervised framework, named cascaded multi-view aggregating network (CMANet) CMANet is superior to state-of-the-art methods in extensive quantitative and qualitative analysis.
  arXiv  Detail & Related papers  (2024-03-19T04:54:59Z)
• MUC: Mixture of Uncalibrated Cameras for Robust 3D Human Body Reconstruction [12.942635715952525]
  Multiple cameras can provide comprehensive multi-view video coverage of a person. Previous studies have overlooked the challenges posed by self-occlusion under multiple views. We introduce a method to reconstruct the 3D human body from multiple uncalibrated camera views.
  arXiv  Detail & Related papers  (2024-03-08T05:03:25Z)
• PF-LRM: Pose-Free Large Reconstruction Model for Joint Pose and Shape Prediction [77.89935657608926]
  We propose a Pose-Free Large Reconstruction Model (PF-LRM) for reconstructing a 3D object from a few unposed images. PF-LRM simultaneously estimates the relative camera poses in 1.3 seconds on a single A100 GPU.
  arXiv  Detail & Related papers  (2023-11-20T18:57:55Z)
• Scene-Aware 3D Multi-Human Motion Capture from a Single Camera [83.06768487435818]
  We consider the problem of estimating the 3D position of multiple humans in a scene as well as their body shape and articulation from a single RGB video recorded with a static camera. We leverage recent advances in computer vision using large-scale pre-trained models for a variety of modalities, including 2D body joints, joint angles, normalized disparity maps, and human segmentation masks. In particular, we estimate the scene depth and unique person scale from normalized disparity predictions using the 2D body joints and joint angles.
  arXiv  Detail & Related papers  (2023-01-12T18:01:28Z)
• Learning Temporal 3D Human Pose Estimation with Pseudo-Labels [3.0954251281114513]
  We present a simple, yet effective, approach for self-supervised 3D human pose estimation. We rely on triangulating 2D body pose estimates of a multiple-view camera system. Our method achieves state-of-the-art performance in the Human3.6M and MPI-INF-3DHP benchmarks.
  arXiv  Detail & Related papers  (2021-10-14T17:40:45Z)
• Self-Supervised 3D Human Pose Estimation with Multiple-View Geometry [2.7541825072548805]
  We present a self-supervised learning algorithm for 3D human pose estimation of a single person based on a multiple-view camera system. We propose a four-loss function learning algorithm, which does not require any 2D or 3D body pose ground-truth.
  arXiv  Detail & Related papers  (2021-08-17T17:31:24Z)
• VoxelTrack: Multi-Person 3D Human Pose Estimation and Tracking in the Wild [98.69191256693703]
  We present VoxelTrack for multi-person 3D pose estimation and tracking from a few cameras which are separated by wide baselines. It employs a multi-branch network to jointly estimate 3D poses and re-identification (Re-ID) features for all people in the environment. It outperforms the state-of-the-art methods by a large margin on three public datasets including Shelf, Campus and CMU Panoptic.
  arXiv  Detail & Related papers  (2021-08-05T08:35:44Z)
• Monocular Expressive Body Regression through Body-Driven Attention [68.63766976089842]
  We introduce ExPose, which regresses the body, face, and hands, in SMPL-X format, from an RGB image. hands and faces are much smaller than the body, occupying very few image pixels. We observe that body estimation localizes the face and hands reasonably well.
  arXiv  Detail & Related papers  (2020-08-20T16:33:47Z)
• Self-Supervised 3D Human Pose Estimation via Part Guided Novel Image Synthesis [72.34794624243281]
  We propose a self-supervised learning framework to disentangle variations from unlabeled video frames. Our differentiable formalization, bridging the representation gap between the 3D pose and spatial part maps, allows us to operate on videos with diverse camera movements.
  arXiv  Detail & Related papers  (2020-04-09T07:55:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.