Prior-Aware Synthetic Data to the Rescue: Animal Pose Estimation with Very Limited Real Data

• URL: http://arxiv.org/abs/2208.13944v1
• Date: Tue, 30 Aug 2022 01:17:50 GMT
• Title: Prior-Aware Synthetic Data to the Rescue: Animal Pose Estimation with Very Limited Real Data
• Authors: Le Jiang, Shuangjun Liu, Xiangyu Bai, Sarah Ostadabbas
• Abstract summary: We present a data efficient strategy for pose estimation in quadrupeds that requires only a small amount of real images from the target animal. It is confirmed that fine-tuning a backbone network with pretrained weights on generic image datasets such as ImageNet can mitigate the high demand for target animal pose data. We introduce a prior-aware synthetic animal data generation pipeline called PASyn to augment the animal pose data essential for robust pose estimation.
• Score: 18.06492246414256
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Accurately annotated image datasets are essential components for studying animal behaviors from their poses. Compared to the number of species we know and may exist, the existing labeled pose datasets cover only a small portion of them, while building comprehensive large-scale datasets is prohibitively expensive. Here, we present a very data efficient strategy targeted for pose estimation in quadrupeds that requires only a small amount of real images from the target animal. It is confirmed that fine-tuning a backbone network with pretrained weights on generic image datasets such as ImageNet can mitigate the high demand for target animal pose data and shorten the training time by learning the the prior knowledge of object segmentation and keypoint estimation in advance. However, when faced with serious data scarcity (i.e., $<10^2$ real images), the model performance stays unsatisfactory, particularly for limbs with considerable flexibility and several comparable parts. We therefore introduce a prior-aware synthetic animal data generation pipeline called PASyn to augment the animal pose data essential for robust pose estimation. PASyn generates a probabilistically-valid synthetic pose dataset, SynAP, through training a variational generative model on several animated 3D animal models. In addition, a style transfer strategy is utilized to blend the synthetic animal image into the real backgrounds. We evaluate the improvement made by our approach with three popular backbone networks and test their pose estimation accuracy on publicly available animal pose images as well as collected from real animals in a zoo.

Related papers

• ZebraPose: Zebra Detection and Pose Estimation using only Synthetic Data [0.2302001830524133]
  We use synthetic data generated with a 3D simulator to obtain the first synthetic dataset that can be used for both detection and 2D pose estimation of zebras. We extensively train and benchmark our detection and 2D pose estimation models on multiple real-world and synthetic datasets. These experiments show how the models trained from scratch and only with synthetic data can consistently generalize to real-world images of zebras.
  arXiv  Detail & Related papers  (2024-08-20T13:28:37Z)
• Learning the 3D Fauna of the Web [70.01196719128912]
  We develop 3D-Fauna, an approach that learns a pan-category deformable 3D animal model for more than 100 animal species jointly. One crucial bottleneck of modeling animals is the limited availability of training data. We show that prior category-specific attempts fail to generalize to rare species with limited training images.
  arXiv  Detail & Related papers  (2024-01-04T18:32:48Z)
• OmniMotionGPT: Animal Motion Generation with Limited Data [70.35662376853163]
  We introduce AnimalML3D, the first text-animal motion dataset with 1240 animation sequences spanning 36 different animal identities. We are able to generate animal motions with high diversity and fidelity, quantitatively and qualitatively outperforming the results of training human motion generation baselines on animal data.
  arXiv  Detail & Related papers  (2023-11-30T07:14:00Z)
• Animal3D: A Comprehensive Dataset of 3D Animal Pose and Shape [32.11280929126699]
  We propose Animal3D, the first comprehensive dataset for mammal animal 3D pose and shape estimation. Animal3D consists of 3379 images collected from 40 mammal species, high-quality annotations of 26 keypoints, and importantly the pose and shape parameters of the SMAL model. Based on the Animal3D dataset, we benchmark representative shape and pose estimation models at: (1) supervised learning from only the Animal3D data, (2) synthetic to real transfer from synthetically generated images, and (3) fine-tuning human pose and shape estimation models.
  arXiv  Detail & Related papers  (2023-08-22T18:57:07Z)
• Of Mice and Pose: 2D Mouse Pose Estimation from Unlabelled Data and Synthetic Prior [0.7499722271664145]
  We propose an approach for estimating 2D mouse body pose from unlabelled images using a synthetically generated empirical pose prior. We adapt this method to the limb structure of the mouse and generate the empirical prior of 2D poses from a synthetic 3D mouse model. In experiments on a new mouse video dataset, we evaluate the performance of the approach by comparing pose predictions to a manually obtained ground truth.
  arXiv  Detail & Related papers  (2023-07-25T09:31:55Z)
• Pose Recognition in the Wild: Animal pose estimation using Agglomerative Clustering and Contrastive Learning [0.0]
  We introduce a novel architecture that is able to recognize the pose of multiple animals fromunlabelled data. We are able to distinguish between body parts of the animal, based on their visual behavior, instead of the underlying anatomy.
  arXiv  Detail & Related papers  (2021-11-16T07:00:31Z)
• SyDog: A Synthetic Dog Dataset for Improved 2D Pose Estimation [3.411873646414169]
  SyDog is a synthetic dataset of dogs containing ground truth pose and bounding box coordinates. We demonstrate that pose estimation models trained on SyDog achieve better performance than models trained purely on real data.
  arXiv  Detail & Related papers  (2021-07-31T14:34:40Z)
• Cascaded deep monocular 3D human pose estimation with evolutionary training data [76.3478675752847]
  Deep representation learning has achieved remarkable accuracy for monocular 3D human pose estimation. This paper proposes a novel data augmentation method that is scalable for massive amount of training data. Our method synthesizes unseen 3D human skeletons based on a hierarchical human representation and synthesizings inspired by prior knowledge.
  arXiv  Detail & Related papers  (2020-06-14T03:09:52Z)
• Self-Supervised 3D Human Pose Estimation via Part Guided Novel Image Synthesis [72.34794624243281]
  We propose a self-supervised learning framework to disentangle variations from unlabeled video frames. Our differentiable formalization, bridging the representation gap between the 3D pose and spatial part maps, allows us to operate on videos with diverse camera movements.
  arXiv  Detail & Related papers  (2020-04-09T07:55:01Z)
• Transferring Dense Pose to Proximal Animal Classes [83.84439508978126]
  We show that it is possible to transfer the knowledge existing in dense pose recognition for humans, as well as in more general object detectors and segmenters, to the problem of dense pose recognition in other classes. We do this by establishing a DensePose model for the new animal which is also geometrically aligned to humans. We also introduce two benchmark datasets labelled in the manner of DensePose for the class chimpanzee and use them to evaluate our approach.
  arXiv  Detail & Related papers  (2020-02-28T21:43:53Z)
• Deformation-aware Unpaired Image Translation for Pose Estimation on Laboratory Animals [56.65062746564091]
  We aim to capture the pose of neuroscience model organisms, without using any manual supervision, to study how neural circuits orchestrate behaviour. Our key contribution is the explicit and independent modeling of appearance, shape and poses in an unpaired image translation framework. We demonstrate improved pose estimation accuracy on Drosophila melanogaster (fruit fly), Caenorhabditis elegans (worm) and Danio rerio (zebrafish)
  arXiv  Detail & Related papers  (2020-01-23T15:34:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.