SuperAnimal pretrained pose estimation models for behavioral analysis

• URL: http://arxiv.org/abs/2203.07436v4
• Date: Sun, 31 Dec 2023 01:17:27 GMT
• Title: SuperAnimal pretrained pose estimation models for behavioral analysis
• Authors: Shaokai Ye and Anastasiia Filippova and Jessy Lauer and Steffen Schneider and Maxime Vidal and Tian Qiu and Alexander Mathis and Mackenzie Weygandt Mathis
• Abstract summary: Quantification of behavior is critical in applications ranging from neuroscience, veterinary medicine and animal conservation efforts. We present a series of technical innovations that enable a new method, collectively called SuperAnimal, to develop unified foundation models.
• Score: 42.206265576708255
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Quantification of behavior is critical in applications ranging from neuroscience, veterinary medicine and animal conservation efforts. A common key step for behavioral analysis is first extracting relevant keypoints on animals, known as pose estimation. However, reliable inference of poses currently requires domain knowledge and manual labeling effort to build supervised models. We present a series of technical innovations that enable a new method, collectively called SuperAnimal, to develop unified foundation models that can be used on over 45 species, without additional human labels. Concretely, we introduce a method to unify the keypoint space across differently labeled datasets (via our generalized data converter) and for training these diverse datasets in a manner such that they don't catastrophically forget keypoints given the unbalanced inputs (via our keypoint gradient masking and memory replay approaches). These models show excellent performance across six pose benchmarks. Then, to ensure maximal usability for end-users, we demonstrate how to fine-tune the models on differently labeled data and provide tooling for unsupervised video adaptation to boost performance and decrease jitter across frames. If the models are fine-tuned, we show SuperAnimal models are 10-100$\times$ more data efficient than prior transfer-learning-based approaches. We illustrate the utility of our models in behavioral classification in mice and gait analysis in horses. Collectively, this presents a data-efficient solution for animal pose estimation.

Related papers

• A Model Generalization Study in Localizing Indoor Cows with COw LOcalization (COLO) dataset [0.0]
  This study investigates the generalization capabilities of YOLOv8 and YOLOv9 models for cow detection in indoor free-stall barn settings. We explore three key hypotheses: (1) Model generalization is equally influenced by changes in lighting conditions and camera angles; (2) Higher model complexity guarantees better generalization performance; (3) Fine-tuning with custom initial weights trained on relevant tasks always brings advantages to detection tasks.
  arXiv  Detail & Related papers  (2024-07-29T18:49:58Z)
• PoseBench: Benchmarking the Robustness of Pose Estimation Models under Corruptions [57.871692507044344]
  Pose estimation aims to accurately identify anatomical keypoints in humans and animals using monocular images. Current models are typically trained and tested on clean data, potentially overlooking the corruption during real-world deployment. We introduce PoseBench, a benchmark designed to evaluate the robustness of pose estimation models against real-world corruption.
  arXiv  Detail & Related papers  (2024-06-20T14:40:17Z)
• From Forest to Zoo: Great Ape Behavior Recognition with ChimpBehave [0.0]
  We introduce ChimpBehave, a novel dataset featuring over 2 hours of video (approximately 193,000 video frames) of zoo-housed chimpanzees. ChimpBehave meticulously annotated with bounding boxes and behavior labels for action recognition. We benchmark our dataset using a state-of-the-art CNN-based action recognition model.
  arXiv  Detail & Related papers  (2024-05-30T13:11:08Z)
• Model Selection with Model Zoo via Graph Learning [45.30615308692713]
  We introduce TransferGraph, a novel framework that reformulates model selection as a graph learning problem. We demonstrate TransferGraph's effectiveness in capturing essential model-dataset relationships, yielding up to a 32% improvement in correlation between predicted performance and the actual fine-tuning results compared to the state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-04-05T09:50:00Z)
• Hyper-Representations for Pre-Training and Transfer Learning [2.9678808525128813]
  We extend hyper-representations for generative use to sample new model weights as pre-training. Our results indicate the potential of knowledge aggregation from model zoos to new models via hyper-representations.
  arXiv  Detail & Related papers  (2022-07-22T09:01:21Z)
• X-model: Improving Data Efficiency in Deep Learning with A Minimax Model [78.55482897452417]
  We aim at improving data efficiency for both classification and regression setups in deep learning. To take the power of both worlds, we propose a novel X-model. X-model plays a minimax game between the feature extractor and task-specific heads.
  arXiv  Detail & Related papers  (2021-10-09T13:56:48Z)
• STAR: Sparse Transformer-based Action Recognition [61.490243467748314]
  This work proposes a novel skeleton-based human action recognition model with sparse attention on the spatial dimension and segmented linear attention on the temporal dimension of data. Experiments show that our model can achieve comparable performance while utilizing much less trainable parameters and achieve high speed in training and inference.
  arXiv  Detail & Related papers  (2021-07-15T02:53:11Z)
• Zoo-Tuning: Adaptive Transfer from a Zoo of Models [82.9120546160422]
  Zoo-Tuning learns to adaptively transfer the parameters of pretrained models to the target task. We evaluate our approach on a variety of tasks, including reinforcement learning, image classification, and facial landmark detection.
  arXiv  Detail & Related papers  (2021-06-29T14:09:45Z)
• Deformation-aware Unpaired Image Translation for Pose Estimation on Laboratory Animals [56.65062746564091]
  We aim to capture the pose of neuroscience model organisms, without using any manual supervision, to study how neural circuits orchestrate behaviour. Our key contribution is the explicit and independent modeling of appearance, shape and poses in an unpaired image translation framework. We demonstrate improved pose estimation accuracy on Drosophila melanogaster (fruit fly), Caenorhabditis elegans (worm) and Danio rerio (zebrafish)
  arXiv  Detail & Related papers  (2020-01-23T15:34:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.