Related papers: STAF: 3D Human Mesh Recovery from Video with Spatio-Temporal Alignment Fusion

STAF: 3D Human Mesh Recovery from Video with Spatio-Temporal Alignment Fusion

URL: http://arxiv.org/abs/2401.01730v1
Date: Wed, 3 Jan 2024 13:07:14 GMT
Title: STAF: 3D Human Mesh Recovery from Video with Spatio-Temporal Alignment Fusion
Authors: Wei Yao, Hongwen Zhang, Yunlian Sun, and Jinhui Tang
Abstract summary: Existing models usually ignore spatial and temporal information, which might lead to mesh and image misalignment and temporal discontinuity. As a video-based model, it leverages coherence clues from human motion by an attention-based Temporal Coherence Fusion Module. In addition, we propose an Average Pooling Module (APM) to allow the model to focus on the entire input sequence rather than just the target frame.
Score: 35.42718669331158
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Abstract: The recovery of 3D human mesh from monocular images has significantly been developed in recent years. However, existing models usually ignore spatial and temporal information, which might lead to mesh and image misalignment and temporal discontinuity. For this reason, we propose a novel Spatio-Temporal Alignment Fusion (STAF) model. As a video-based model, it leverages coherence clues from human motion by an attention-based Temporal Coherence Fusion Module (TCFM). As for spatial mesh-alignment evidence, we extract fine-grained local information through predicted mesh projection on the feature maps. Based on the spatial features, we further introduce a multi-stage adjacent Spatial Alignment Fusion Module (SAFM) to enhance the feature representation of the target frame. In addition to the above, we propose an Average Pooling Module (APM) to allow the model to focus on the entire input sequence rather than just the target frame. This method can remarkably improve the smoothness of recovery results from video. Extensive experiments on 3DPW, MPII3D, and H36M demonstrate the superiority of STAF. We achieve a state-of-the-art trade-off between precision and smoothness. Our code and more video results are on the project page https://yw0208.github.io/staf/