AiOS: All-in-One-Stage Expressive Human Pose and Shape Estimation

• URL: http://arxiv.org/abs/2403.17934v1
• Date: Tue, 26 Mar 2024 17:59:23 GMT
• Title: AiOS: All-in-One-Stage Expressive Human Pose and Shape Estimation
• Authors: Qingping Sun, Yanjun Wang, Ailing Zeng, Wanqi Yin, Chen Wei, Wenjia Wang, Haiyi Mei, Chi Sing Leung, Ziwei Liu, Lei Yang, Zhongang Cai,
• Abstract summary: We introduce a novel all-in-one-stage framework, AiOS, for expressive human pose and shape recovery without an additional human detection step. We first employ a human token to probe a human location in the image and encode global features for each instance. Then, we introduce a joint-related token to probe the human joint in the image and encoder a fine-grained local feature.
• Score: 55.179287851188036
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Expressive human pose and shape estimation (a.k.a. 3D whole-body mesh recovery) involves the human body, hand, and expression estimation. Most existing methods have tackled this task in a two-stage manner, first detecting the human body part with an off-the-shelf detection model and inferring the different human body parts individually. Despite the impressive results achieved, these methods suffer from 1) loss of valuable contextual information via cropping, 2) introducing distractions, and 3) lacking inter-association among different persons and body parts, inevitably causing performance degradation, especially for crowded scenes. To address these issues, we introduce a novel all-in-one-stage framework, AiOS, for multiple expressive human pose and shape recovery without an additional human detection step. Specifically, our method is built upon DETR, which treats multi-person whole-body mesh recovery task as a progressive set prediction problem with various sequential detection. We devise the decoder tokens and extend them to our task. Specifically, we first employ a human token to probe a human location in the image and encode global features for each instance, which provides a coarse location for the later transformer block. Then, we introduce a joint-related token to probe the human joint in the image and encoder a fine-grained local feature, which collaborates with the global feature to regress the whole-body mesh. This straightforward but effective model outperforms previous state-of-the-art methods by a 9% reduction in NMVE on AGORA, a 30% reduction in PVE on EHF, a 10% reduction in PVE on ARCTIC, and a 3% reduction in PVE on EgoBody.

Related papers

• Closely Interactive Human Reconstruction with Proxemics and Physics-Guided Adaption [64.07607726562841]
  Existing multi-person human reconstruction approaches mainly focus on recovering accurate poses or avoiding penetration. In this work, we tackle the task of reconstructing closely interactive humans from a monocular video. We propose to leverage knowledge from proxemic behavior and physics to compensate the lack of visual information.
  arXiv  Detail & Related papers  (2024-04-17T11:55:45Z)
• Probabilistic Human Mesh Recovery in 3D Scenes from Egocentric Views [32.940614931864154]
  We propose a scene-conditioned diffusion method to model the body pose distribution. The method generates bodies in plausible human-scene interactions. It achieves superior accuracy for visible joints and diversity for invisible body parts.
  arXiv  Detail & Related papers  (2023-04-12T17:58:57Z)
• Explicit Occlusion Reasoning for Multi-person 3D Human Pose Estimation [33.86986028882488]
  Occlusion poses a great threat to monocular multi-person 3D human pose estimation due to large variability in terms of the shape, appearance, and position of occluders. Existing methods try to handle occlusion with pose priors/constraints, data augmentation, or implicit reasoning. We develop a method to explicitly model this process that significantly improves bottom-up multi-person human pose estimation.
  arXiv  Detail & Related papers  (2022-07-29T22:12:50Z)
• KTN: Knowledge Transfer Network for Learning Multi-person 2D-3D Correspondences [77.56222946832237]
  We present a novel framework to detect the densepose of multiple people in an image. The proposed method, which we refer to Knowledge Transfer Network (KTN), tackles two main problems. It simultaneously maintains feature resolution and suppresses background pixels, and this strategy results in substantial increase in accuracy.
  arXiv  Detail & Related papers  (2022-06-21T03:11:37Z)
• Dual networks based 3D Multi-Person Pose Estimation from Monocular Video [42.01876518017639]
  Multi-person 3D pose estimation is more challenging than single pose estimation. Existing top-down and bottom-up approaches to pose estimation suffer from detection errors. We propose the integration of top-down and bottom-up approaches to exploit their strengths.
  arXiv  Detail & Related papers  (2022-05-02T08:53:38Z)
• Monocular 3D Multi-Person Pose Estimation by Integrating Top-Down and Bottom-Up Networks [33.974241749058585]
  Multi-person pose estimation can cause human detection to be erroneous and human-joints grouping to be unreliable. Existing top-down methods rely on human detection and thus suffer from these problems. We propose the integration of top-down and bottom-up approaches to exploit their strengths.
  arXiv  Detail & Related papers  (2021-04-05T07:05:21Z)
• Neural Descent for Visual 3D Human Pose and Shape [67.01050349629053]
  We present deep neural network methodology to reconstruct the 3d pose and shape of people, given an input RGB image. We rely on a recently introduced, expressivefull body statistical 3d human model, GHUM, trained end-to-end. Central to our methodology, is a learning to learn and optimize approach, referred to as HUmanNeural Descent (HUND), which avoids both second-order differentiation.
  arXiv  Detail & Related papers  (2020-08-16T13:38:41Z)
• Coherent Reconstruction of Multiple Humans from a Single Image [68.3319089392548]
  In this work, we address the problem of multi-person 3D pose estimation from a single image. A typical regression approach in the top-down setting of this problem would first detect all humans and then reconstruct each one of them independently. Our goal is to train a single network that learns to avoid these problems and generate a coherent 3D reconstruction of all the humans in the scene.
  arXiv  Detail & Related papers  (2020-06-15T17:51:45Z)
• HEMlets PoSh: Learning Part-Centric Heatmap Triplets for 3D Human Pose and Shape Estimation [60.35776484235304]
  This work attempts to address the uncertainty of lifting the detected 2D joints to the 3D space by introducing an intermediate state-Part-Centric Heatmap Triplets (HEMlets) The HEMlets utilize three joint-heatmaps to represent the relative depth information of the end-joints for each skeletal body part. A Convolutional Network (ConvNet) is first trained to predict HEMlets from the input image, followed by a volumetric joint-heatmap regression.
  arXiv  Detail & Related papers  (2020-03-10T04:03:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.