Beyond Static Frames: Temporal Aggregate-and-Restore Vision Transformer for Human Pose Estimation

• URL: http://arxiv.org/abs/2603.05929v1
• Date: Fri, 06 Mar 2026 05:45:04 GMT
• Title: Beyond Static Frames: Temporal Aggregate-and-Restore Vision Transformer for Human Pose Estimation
• Abstract summary: Vision Transformers (ViTs) have recently achieved state-of-the-art performance in 2D human pose estimation.<n> TAR-ViTPose enhances static ViT representations by aggregating temporal cues across frames in a plug-and-play manner.<n>Our approach outperforms existing state-of-the-art video-based methods, while also achieving a noticeably higher runtime frame rate in real-world applications.
• Score: 7.63512442757446
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision Transformers (ViTs) have recently achieved state-of-the-art performance in 2D human pose estimation due to their strong global modeling capability. However, existing ViT-based pose estimators are designed for static images and process each frame independently, thereby ignoring the temporal coherence that exists in video sequences. This limitation often results in unstable predictions, especially in challenging scenes involving motion blur, occlusion, or defocus. In this paper, we propose TAR-ViTPose, a novel Temporal Aggregate-and-Restore Vision Transformer tailored for video-based 2D human pose estimation. TAR-ViTPose enhances static ViT representations by aggregating temporal cues across frames in a plug-and-play manner, leading to more robust and accurate pose estimation. To effectively aggregate joint-specific features that are temporally aligned across frames, we introduce a joint-centric temporal aggregation (JTA) that assigns each joint a learnable query token to selectively attend to its corresponding regions from neighboring frames. Furthermore, we develop a global restoring attention (GRA) to restore the aggregated temporal features back into the token sequence of the current frame, enriching its pose representation while fully preserving global context for precise keypoint localization. Extensive experiments demonstrate that TAR-ViTPose substantially improves upon the single-frame baseline ViTPose, achieving a +2.3 mAP gain on the PoseTrack2017 benchmark. Moreover, our approach outperforms existing state-of-the-art video-based methods, while also achieving a noticeably higher runtime frame rate in real-world applications. Project page: https://github.com/zgspose/TARViTPose.

Related papers

• CineOrchestra: Unified Entity-Centric Conditioning for Cinematic Video Generation [57.47993858752813]
  We present CineOrchestra, a unified video diffusion model that controls subjects, events, cameras, and shot transitions simultaneously.<n>Our key insight is that these heterogeneous cinematic elements share a fundamental structure.<n>CineOrchestra outperforms six per-axis specialists on dense caption following and shot-transition timing.
  arXiv  Detail & Related papers  (2026-06-11T17:59:10Z)
• StableDPT: Temporal Stable Monocular Video Depth Estimation [14.453483279783908]
  We propose a novel approach that adapts any state-of-the-art image-based (depth) estimation model for video processing.<n>Our architecture builds upon an off-the-shelf Vision Transformer (ViT) encoder and enhances the Dense Prediction Transformer (DPT) head.<n> Evaluations on multiple benchmark datasets demonstrate improved temporal consistency, competitive state-of-the-art performance and on top 2x faster processing in real-world scenarios.
  arXiv  Detail & Related papers  (2026-01-06T08:02:14Z)
• Learning Spatio-Temporal Feature Representations for Video-Based Gaze Estimation [50.05866669110754]
  Video-based gaze estimation methods aim to capture the inherently temporal dynamics of human eye gaze from multiple image frames.<n>We propose the Spatio-Temporal Gaze Network (ST-Gaze), a model that combines a CNN backbone with dedicated channel attention and self-attention modules.<n>We show that ST-Gaze achieves state-of-the-art performance both with and without person-specific adaptation.
  arXiv  Detail & Related papers  (2025-12-19T15:15:58Z)
• End-to-End Multi-Person Pose Estimation with Pose-Aware Video Transformer [7.19764062839405]
  We present a fully end-to-end framework for multi-person 2D pose estimation in videos.<n>A key challenge is to associate individuals across frames under complex and overlapping temporal trajectories.<n>We introduce a novel Pose-Aware VideoErEr Network (PAVE-Net), which features a spatial encoder to model intra-frame relations and atemporal decoder pose.
  arXiv  Detail & Related papers  (2025-11-17T10:19:35Z)
• Towards Robust and Generalizable Continuous Space-Time Video Super-Resolution with Events [71.2439653098351]
  Continuous space-time video super-STVSR has garnered increasing interest for its capability to reconstruct high-resolution and high-frame-rate videos at arbitrary temporal scales.<n>We present EvEnhancer, a novel approach that marries unique properties of high temporal and high dynamic range encapsulated in event streams.<n>Our method achieves state-of-the-art performance on both synthetic and real-world datasets, while maintaining generalizability at OOD scales.
  arXiv  Detail & Related papers  (2025-10-04T15:23:07Z)
• Tracktention: Leveraging Point Tracking to Attend Videos Faster and Better [61.381599921020175]
  Temporal consistency is critical in video prediction to ensure that outputs are coherent and free of artifacts.<n>Traditional methods, such as temporal attention and 3D convolution, may struggle with significant object motion.<n>We propose the Tracktention Layer, a novel architectural component that explicitly integrates motion information using point tracks.
  arXiv  Detail & Related papers  (2025-03-25T17:58:48Z)
• SpatioTemporal Learning for Human Pose Estimation in Sparsely-Labeled Videos [18.37601213802529]
  STDPose is a novel framework that enhances human pose estimation by learning in sparsely-labeled videos.<n>STDPose establishes a new benchmark for both video pose propagation (i.e., propagating pose from labeled frames to unlabeled frames) and pose estimation tasks.
  arXiv  Detail & Related papers  (2025-01-25T04:43:12Z)
• Poseidon: A ViT-based Architecture for Multi-Frame Pose Estimation with Adaptive Frame Weighting and Multi-Scale Feature Fusion [43.59385149982744]
  Single-frame pose estimation has seen significant progress, but it often fails to capture the temporal dynamics for understanding complex, continuous movements.<n>We propose Poseidon, a novel multi-frame pose estimation architecture that extends the ViTPose model by integrating temporal information.<n>Our approach achieves state-of-the-art performance on the PoseTrack21 and PoseTrack18 datasets, achieving mAP scores of 88.3 and 87.8, respectively.
  arXiv  Detail & Related papers  (2025-01-14T21:34:34Z)
• TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinement [64.11385310305612]
  We present a novel model for Tracking Any Point (TAP) that effectively tracks any queried point on any physical surface throughout a video sequence. Our approach employs two stages: (1) a matching stage, which independently locates a suitable candidate point match for the query point on every other frame, and (2) a refinement stage, which updates both the trajectory and query features based on local correlations. The resulting model surpasses all baseline methods by a significant margin on the TAP-Vid benchmark, as demonstrated by an approximate 20% absolute average Jaccard (AJ) improvement on DAVIS.
  arXiv  Detail & Related papers  (2023-06-14T17:07:51Z)
• A Unified Framework for Event-based Frame Interpolation with Ad-hoc Deblurring in the Wild [72.0226493284814]
  We propose a unified framework for event-based frame that performs deblurring ad-hoc.<n>Our network consistently outperforms previous state-of-the-art methods on frame, single image deblurring, and the joint task of both.
  arXiv  Detail & Related papers  (2023-01-12T18:19:00Z)
• Stand-Alone Inter-Frame Attention in Video Models [164.06137994796487]
  We present a new recipe of inter-frame attention block, namely Stand-alone Inter-temporal Attention (SIFA) SIFA remoulds the deformable design via re-scaling the offset predictions by the difference between two frames. We further plug SIFA block into ConvNets and Vision Transformer, respectively, to devise SIFA-Net and SIFA-Transformer.
  arXiv  Detail & Related papers  (2022-06-14T15:51:28Z)
• Temporal Feature Alignment and Mutual Information Maximization for Video-Based Human Pose Estimation [38.571715193347366]
  We present a novel hierarchical alignment framework for multi-frame human pose estimation. We rank No.1 in the Multi-frame Person Pose Estimation Challenge on benchmark dataset PoseTrack 2017, and obtain state-of-the-art performance on benchmarks Sub-JHMDB and Pose-Track 2018.
  arXiv  Detail & Related papers  (2022-03-29T04:29:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.