Global-to-Local Modeling for Video-based 3D Human Pose and Shape Estimation

• URL: http://arxiv.org/abs/2303.14747v1
• Date: Sun, 26 Mar 2023 14:57:49 GMT
• Title: Global-to-Local Modeling for Video-based 3D Human Pose and Shape Estimation
• Authors: Xiaolong Shen, Zongxin Yang, Xiaohan Wang, Jianxin Ma, Chang Zhou, Yi Yang
• Abstract summary: Video-based 3D human pose and shape estimations are evaluated by intra-frame accuracy and inter-frame smoothness. We propose to structurally decouple the modeling of long-term and short-term correlations in an end-to-end framework, Global-to-Local Transformer (GLoT) Our GLoT surpasses previous state-of-the-art methods with the lowest model parameters on popular benchmarks, i.e., 3DPW, MPI-INF-3DHP, and Human3.6M.
• Score: 53.04781510348416
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Video-based 3D human pose and shape estimations are evaluated by intra-frame accuracy and inter-frame smoothness. Although these two metrics are responsible for different ranges of temporal consistency, existing state-of-the-art methods treat them as a unified problem and use monotonous modeling structures (e.g., RNN or attention-based block) to design their networks. However, using a single kind of modeling structure is difficult to balance the learning of short-term and long-term temporal correlations, and may bias the network to one of them, leading to undesirable predictions like global location shift, temporal inconsistency, and insufficient local details. To solve these problems, we propose to structurally decouple the modeling of long-term and short-term correlations in an end-to-end framework, Global-to-Local Transformer (GLoT). First, a global transformer is introduced with a Masked Pose and Shape Estimation strategy for long-term modeling. The strategy stimulates the global transformer to learn more inter-frame correlations by randomly masking the features of several frames. Second, a local transformer is responsible for exploiting local details on the human mesh and interacting with the global transformer by leveraging cross-attention. Moreover, a Hierarchical Spatial Correlation Regressor is further introduced to refine intra-frame estimations by decoupled global-local representation and implicit kinematic constraints. Our GLoT surpasses previous state-of-the-art methods with the lowest model parameters on popular benchmarks, i.e., 3DPW, MPI-INF-3DHP, and Human3.6M. Codes are available at https://github.com/sxl142/GLoT.

Related papers

• PoseMamba: Monocular 3D Human Pose Estimation with Bidirectional Global-Local Spatio-Temporal State Space Model [7.286873011001679]
  We propose a purely SSM-based approach with linear correlations for complexityD human pose estimation in monocular video video. Specifically, we propose a bidirectional global temporal-local-temporal block that comprehensively models human joint relations within individual frames as well as across frames. This strategy provides a more logical geometric ordering strategy, resulting in a combined-local spatial scan.
  arXiv  Detail & Related papers  (2024-08-07T04:38:03Z)
• Boosting Cross-Domain Point Classification via Distilling Relational Priors from 2D Transformers [59.0181939916084]
  Traditional 3D networks mainly focus on local geometric details and ignore the topological structure between local geometries. We propose a novel Priors Distillation (RPD) method to extract priors from the well-trained transformers on massive images. Experiments on the PointDA-10 and the Sim-to-Real datasets verify that the proposed method consistently achieves the state-of-the-art performance of UDA for point cloud classification.
  arXiv  Detail & Related papers  (2024-07-26T06:29:09Z)
• Graph and Skipped Transformer: Exploiting Spatial and Temporal Modeling Capacities for Efficient 3D Human Pose Estimation [36.93661496405653]
  We take a global approach to exploit Transformer-temporal information with a concise Graph and Skipped Transformer architecture. Specifically, in 3D pose stage, coarse-grained body parts are deployed to construct a fully data-driven adaptive model. Experiments are conducted on Human3.6M, MPI-INF-3DHP and Human-Eva benchmarks.
  arXiv  Detail & Related papers  (2024-07-03T10:42:09Z)
• Double-chain Constraints for 3D Human Pose Estimation in Images and Videos [21.42410292863492]
  Reconstructing 3D poses from 2D poses lacking depth information is challenging due to the complexity and diversity of human motion. We propose a novel model, called Double-chain Graph Convolutional Transformer (DC-GCT), to constrain the pose. We show that DC-GCT achieves state-of-the-art performance on two challenging datasets.
  arXiv  Detail & Related papers  (2023-08-10T02:41:18Z)
• Hierarchical Local-Global Transformer for Temporal Sentence Grounding [58.247592985849124]
  This paper studies the multimedia problem of temporal sentence grounding. It aims to accurately determine the specific video segment in an untrimmed video according to a given sentence query.
  arXiv  Detail & Related papers  (2022-08-31T14:16:56Z)
• Back to MLP: A Simple Baseline for Human Motion Prediction [59.18776744541904]
  This paper tackles the problem of human motion prediction, consisting in forecasting future body poses from historically observed sequences. We show that the performance of these approaches can be surpassed by a light-weight and purely architectural architecture with only 0.14M parameters. An exhaustive evaluation on Human3.6M, AMASS and 3DPW datasets shows that our method, which we dub siMLPe, consistently outperforms all other approaches.
  arXiv  Detail & Related papers  (2022-07-04T16:35:58Z)
• CrossFormer: Cross Spatio-Temporal Transformer for 3D Human Pose Estimation [24.08170512746056]
  3D human pose estimation can be handled by encoding the geometric dependencies between the body parts and enforcing the kinematic constraints. Recent Transformer has been adopted to encode the long-range dependencies between the joints in the spatial and temporal domains. We propose a novel pose estimation Transformer featuring rich representations of body joints critical for capturing subtle changes across frames.
  arXiv  Detail & Related papers  (2022-03-24T23:40:11Z)
• MixSTE: Seq2seq Mixed Spatio-Temporal Encoder for 3D Human Pose Estimation in Video [75.23812405203778]
  Recent solutions have been introduced to estimate 3D human pose from 2D keypoint sequence by considering body joints among all frames globally to learn-temporal correlation. We propose Mix Mix, which has temporal transformer block to separately model the temporal motion of each joint and a transformer block inter-joint spatial correlation. In addition, the network output is extended from the central frame to entire frames of input video, improving the coherence between the input and output benchmarks.
  arXiv  Detail & Related papers  (2022-03-02T04:20:59Z)
• Global Filter Networks for Image Classification [90.81352483076323]
  We present a conceptually simple yet computationally efficient architecture that learns long-term spatial dependencies in the frequency domain with log-linear complexity. Our results demonstrate that GFNet can be a very competitive alternative to transformer-style models and CNNs in efficiency, generalization ability and robustness.
  arXiv  Detail & Related papers  (2021-07-01T17:58:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.