Hourglass Tokenizer for Efficient Transformer-Based 3D Human Pose Estimation

• URL: http://arxiv.org/abs/2311.12028v2
• Date: Wed, 27 Mar 2024 11:43:28 GMT
• Title: Hourglass Tokenizer for Efficient Transformer-Based 3D Human Pose Estimation
• Authors: Wenhao Li, Mengyuan Liu, Hong Liu, Pichao Wang, Jialun Cai, Nicu Sebe,
• Abstract summary: We present a plug-and-play pruning-and-recovering framework, called Hourglass Tokenizer (HoT), for efficient transformer-based 3D pose estimation from videos. Our HoDT begins with pruning pose tokens of redundant frames and ends with recovering full-length tokens, resulting in a few pose tokens in the intermediate transformer blocks. Our method can achieve both high efficiency and estimation accuracy compared to the original VPT models.
• Score: 73.31524865643709
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Transformers have been successfully applied in the field of video-based 3D human pose estimation. However, the high computational costs of these video pose transformers (VPTs) make them impractical on resource-constrained devices. In this paper, we present a plug-and-play pruning-and-recovering framework, called Hourglass Tokenizer (HoT), for efficient transformer-based 3D human pose estimation from videos. Our HoT begins with pruning pose tokens of redundant frames and ends with recovering full-length tokens, resulting in a few pose tokens in the intermediate transformer blocks and thus improving the model efficiency. To effectively achieve this, we propose a token pruning cluster (TPC) that dynamically selects a few representative tokens with high semantic diversity while eliminating the redundancy of video frames. In addition, we develop a token recovering attention (TRA) to restore the detailed spatio-temporal information based on the selected tokens, thereby expanding the network output to the original full-length temporal resolution for fast inference. Extensive experiments on two benchmark datasets (i.e., Human3.6M and MPI-INF-3DHP) demonstrate that our method can achieve both high efficiency and estimation accuracy compared to the original VPT models. For instance, applying to MotionBERT and MixSTE on Human3.6M, our HoT can save nearly 50% FLOPs without sacrificing accuracy and nearly 40% FLOPs with only 0.2% accuracy drop, respectively. Code and models are available at https://github.com/NationalGAILab/HoT.

Related papers

• Efficient Point Transformer with Dynamic Token Aggregating for Point Cloud Processing [19.73918716354272]
  We propose an efficient point TransFormer with Dynamic Token Aggregating (DTA-Former) for point cloud representation and processing. It achieves SOTA performance with up to 30$times$ faster than prior point Transformers on ModelNet40, ShapeNet, and airborne MultiSpectral LiDAR (MS-LiDAR) datasets.
  arXiv  Detail & Related papers  (2024-05-23T20:50:50Z)
• Efficient Video Action Detection with Token Dropout and Context Refinement [67.10895416008911]
  We propose an end-to-end framework for efficient video action detection (ViTs) In a video clip, we maintain tokens from its perspective while preserving tokens relevant to actor motions from other frames. Second, we refine scene context by leveraging remaining tokens for better recognizing actor identities.
  arXiv  Detail & Related papers  (2023-04-17T17:21:21Z)
• Capturing the motion of every joint: 3D human pose and shape estimation with independent tokens [34.50928515515274]
  We present a novel method to estimate 3D human pose and shape from monocular videos. The proposed method attains superior performances on the 3DPW and Human3.6M datasets.
  arXiv  Detail & Related papers  (2023-03-01T07:48:01Z)
• SVFormer: Semi-supervised Video Transformer for Action Recognition [88.52042032347173]
  We introduce SVFormer, which adopts a steady pseudo-labeling framework to cope with unlabeled video samples. In addition, we propose a temporal warping to cover the complex temporal variation in videos. In particular, SVFormer outperforms the state-of-the-art by 31.5% with fewer training epochs under the 1% labeling rate of Kinetics-400.
  arXiv  Detail & Related papers  (2022-11-23T18:58:42Z)
• Uplift and Upsample: Efficient 3D Human Pose Estimation with Uplifting Transformers [28.586258731448687]
  We present a Transformer-based pose uplifting scheme that can operate on temporally sparse 2D pose sequences. We show how masked token modeling can be utilized for temporal upsampling within Transformer blocks. We evaluate our method on two popular benchmark datasets: Human3.6M and MPI-INF-3DHP.
  arXiv  Detail & Related papers  (2022-10-12T12:00:56Z)
• SaiT: Sparse Vision Transformers through Adaptive Token Pruning [5.1477382898520485]
  Sparse adaptive image Transformer (SaiT) offers varying levels of model acceleration by merely changing the token sparsity on the fly. SaiT reduces the complexity (FLOPs) by 39% - 43% and increases the throughput by 67% - 91% with less than 0.5% accuracy loss.
  arXiv  Detail & Related papers  (2022-10-11T23:26:42Z)
• Pix4Point: Image Pretrained Standard Transformers for 3D Point Cloud Understanding [62.502694656615496]
  We present Progressive Point Patch Embedding and present a new point cloud Transformer model namely PViT. PViT shares the same backbone as Transformer but is shown to be less hungry for data, enabling Transformer to achieve performance comparable to the state-of-the-art. We formulate a simple yet effective pipeline dubbed "Pix4Point" that allows harnessing Transformers pretrained in the image domain to enhance downstream point cloud understanding.
  arXiv  Detail & Related papers  (2022-08-25T17:59:29Z)
• Spatiotemporal Self-attention Modeling with Temporal Patch Shift for Action Recognition [34.98846882868107]
  We propose a Temporal Patch Shift (TPS) method for efficient 3D self-attention modeling in transformers for video-based action recognition. As a result, we can compute 3D self-attention memory using nearly the same complexity and cost as 2D self-attention.
  arXiv  Detail & Related papers  (2022-07-27T02:47:07Z)
• Long-Short Temporal Contrastive Learning of Video Transformers [62.71874976426988]
  Self-supervised pretraining of video transformers on video-only datasets can lead to action recognition results on par or better than those obtained with supervised pretraining on large-scale image datasets. Our approach, named Long-Short Temporal Contrastive Learning, enables video transformers to learn an effective clip-level representation by predicting temporal context captured from a longer temporal extent.
  arXiv  Detail & Related papers  (2021-06-17T02:30:26Z)
• ViViT: A Video Vision Transformer [75.74690759089529]
  We present pure-transformer based models for video classification. Our model extracts-temporal tokens from the input video, which are then encoded by a series of transformer layers. We show how we can effectively regularise the model during training and leverage pretrained image models to be able to train on comparatively small datasets.
  arXiv  Detail & Related papers  (2021-03-29T15:27:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.