Self-supervised Keypoint Correspondences for Multi-Person Pose Estimation and Tracking in Videos

• URL: http://arxiv.org/abs/2004.12652v3
• Date: Mon, 15 Mar 2021 11:48:44 GMT
• Title: Self-supervised Keypoint Correspondences for Multi-Person Pose Estimation and Tracking in Videos
• Authors: Umer Rafi, Andreas Doering, Bastian Leibe, Juergen Gall
• Abstract summary: We propose an approach that relies on keypoint correspondences for associating persons in videos. Instead of training the network for estimating keypoint correspondences on video data, it is trained on a large scale image datasets for human pose estimation. Our approach achieves state-of-the-art results for multi-frame pose estimation and multi-person pose tracking on the PosTrack $2017$ and PoseTrack $2018$ data sets.
• Score: 32.43899916477434
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Video annotation is expensive and time consuming. Consequently, datasets for multi-person pose estimation and tracking are less diverse and have more sparse annotations compared to large scale image datasets for human pose estimation. This makes it challenging to learn deep learning based models for associating keypoints across frames that are robust to nuisance factors such as motion blur and occlusions for the task of multi-person pose tracking. To address this issue, we propose an approach that relies on keypoint correspondences for associating persons in videos. Instead of training the network for estimating keypoint correspondences on video data, it is trained on a large scale image datasets for human pose estimation using self-supervision. Combined with a top-down framework for human pose estimation, we use keypoints correspondences to (i) recover missed pose detections (ii) associate pose detections across video frames. Our approach achieves state-of-the-art results for multi-frame pose estimation and multi-person pose tracking on the PosTrack $2017$ and PoseTrack $2018$ data sets.

Related papers

• SRPose: Two-view Relative Pose Estimation with Sparse Keypoints [51.49105161103385]
  SRPose is a sparse keypoint-based framework for two-view relative pose estimation in camera-to-world and object-to-camera scenarios. It achieves competitive or superior performance compared to state-of-the-art methods in terms of accuracy and speed. It is robust to different image sizes and camera intrinsics, and can be deployed with low computing resources.
  arXiv  Detail & Related papers  (2024-07-11T05:46:35Z)
• Reconstructing Close Human Interactions from Multiple Views [38.924950289788804]
  This paper addresses the challenging task of reconstructing the poses of multiple individuals engaged in close interactions, captured by multiple calibrated cameras. We introduce a novel system to address these challenges. Our system integrates a learning-based pose estimation component and its corresponding training and inference strategies.
  arXiv  Detail & Related papers  (2024-01-29T14:08:02Z)
• Pose for Everything: Towards Category-Agnostic Pose Estimation [93.07415325374761]
  Category-Agnostic Pose Estimation (CAPE) aims to create a pose estimation model capable of detecting the pose of any class of object given only a few samples with keypoint definition. A transformer-based Keypoint Interaction Module (KIM) is proposed to capture both the interactions among different keypoints and the relationship between the support and query images. We also introduce Multi-category Pose (MP-100) dataset, which is a 2D pose dataset of 100 object categories containing over 20K instances and is well-designed for developing CAPE algorithms.
  arXiv  Detail & Related papers  (2022-07-21T09:40:54Z)
• Learning Dynamics via Graph Neural Networks for Human Pose Estimation and Tracking [98.91894395941766]
  We propose a novel online approach to learning the pose dynamics, which are independent of pose detections in current fame. Specifically, we derive this prediction of dynamics through a graph neural network(GNN) that explicitly accounts for both spatial-temporal and visual information. Experiments on PoseTrack 2017 and PoseTrack 2018 datasets demonstrate that the proposed method achieves results superior to the state of the art on both human pose estimation and tracking tasks.
  arXiv  Detail & Related papers  (2021-06-07T16:36:50Z)
• Deep Dual Consecutive Network for Human Pose Estimation [44.41818683253614]
  We propose a novel multi-frame human pose estimation framework, leveraging abundant temporal cues between video frames to facilitate keypoint detection. Our method ranks No.1 in the Multi-frame Person Pose Challenge Challenge on the large-scale benchmark datasets PoseTrack 2017 and PoseTrack 2018.
  arXiv  Detail & Related papers  (2021-03-12T13:11:27Z)
• PoseTrackReID: Dataset Description [97.7241689753353]
  Pose information is helpful to disentangle useful feature information from background or occlusion noise. With PoseTrackReID, we want to bridge the gap between person re-ID and multi-person pose tracking. This dataset provides a good benchmark for current state-of-the-art methods on multi-frame person re-ID.
  arXiv  Detail & Related papers  (2020-11-12T07:44:25Z)
• Towards Accurate Human Pose Estimation in Videos of Crowded Scenes [134.60638597115872]
  We focus on improving human pose estimation in videos of crowded scenes from the perspectives of exploiting temporal context and collecting new data. For one frame, we forward the historical poses from the previous frames and backward the future poses from the subsequent frames to current frame, leading to stable and accurate human pose estimation in videos. In this way, our model achieves best performance on 7 out of 13 videos and 56.33 average w_AP on test dataset of HIE challenge.
  arXiv  Detail & Related papers  (2020-10-16T13:19:11Z)
• Self-Supervised 3D Human Pose Estimation via Part Guided Novel Image Synthesis [72.34794624243281]
  We propose a self-supervised learning framework to disentangle variations from unlabeled video frames. Our differentiable formalization, bridging the representation gap between the 3D pose and spatial part maps, allows us to operate on videos with diverse camera movements.
  arXiv  Detail & Related papers  (2020-04-09T07:55:01Z)
• Deep Reinforcement Learning for Active Human Pose Estimation [35.229529080763925]
  We introduce Pose-DRL, a fully trainable deep reinforcement learning-based active pose estimation architecture. We show that our model learns to select viewpoints that yield significantly more accurate pose estimates compared to strong multi-view baselines.
  arXiv  Detail & Related papers  (2020-01-07T13:35:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.