SemiMultiPose: A Semi-supervised Multi-animal Pose Estimation Framework
- URL: http://arxiv.org/abs/2204.07072v1
- Date: Thu, 14 Apr 2022 16:06:55 GMT
- Title: SemiMultiPose: A Semi-supervised Multi-animal Pose Estimation Framework
- Authors: Ari Blau, Christoph Gebhardt, Andres Bendesky, Liam Paninski, and Anqi
Wu
- Abstract summary: Multi-animal pose estimation is essential for studying animals' social behaviors in neuroscience and neuroethology.
We propose a novel semi-supervised architecture for multi-animal pose estimation, leveraging the pervasive structures in unlabeled frames in behavior videos.
The resulting algorithm will provide superior multi-animal pose estimation results on three animal experiments.
- Score: 10.523555645910255
- License: http://creativecommons.org/licenses/by-nc-nd/4.0/
- Abstract: Multi-animal pose estimation is essential for studying animals' social
behaviors in neuroscience and neuroethology. Advanced approaches have been
proposed to support multi-animal estimation and achieve state-of-the-art
performance. However, these models rarely exploit unlabeled data during
training even though real world applications have exponentially more unlabeled
frames than labeled frames. Manually adding dense annotations for a large
number of images or videos is costly and labor-intensive, especially for
multiple instances. Given these deficiencies, we propose a novel
semi-supervised architecture for multi-animal pose estimation, leveraging the
abundant structures pervasive in unlabeled frames in behavior videos to enhance
training, which is critical for sparsely-labeled problems. The resulting
algorithm will provide superior multi-animal pose estimation results on three
animal experiments compared to the state-of-the-art baseline and exhibits more
predictive power in sparsely-labeled data regimes.
Related papers
- PoseBench: Benchmarking the Robustness of Pose Estimation Models under Corruptions [57.871692507044344]
Pose estimation aims to accurately identify anatomical keypoints in humans and animals using monocular images.
Current models are typically trained and tested on clean data, potentially overlooking the corruption during real-world deployment.
We introduce PoseBench, a benchmark designed to evaluate the robustness of pose estimation models against real-world corruption.
arXiv Detail & Related papers (2024-06-20T14:40:17Z) - Reviving the Context: Camera Trap Species Classification as Link Prediction on Multimodal Knowledge Graphs [31.22129440376567]
We exploit the structured context linked to camera trap images to boost out-of-distribution generalization for species classification tasks in camera traps.
A picture of a wild animal could be linked to details about the time and place it was captured, as well as structured biological knowledge about the animal species.
We propose a novel framework that transforms species classification as link prediction in a multimodal knowledge graph.
arXiv Detail & Related papers (2023-12-31T23:32:03Z) - Multimodal Foundation Models for Zero-shot Animal Species Recognition in
Camera Trap Images [57.96659470133514]
Motion-activated camera traps constitute an efficient tool for tracking and monitoring wildlife populations across the globe.
Supervised learning techniques have been successfully deployed to analyze such imagery, however training such techniques requires annotations from experts.
Reducing the reliance on costly labelled data has immense potential in developing large-scale wildlife tracking solutions with markedly less human labor.
arXiv Detail & Related papers (2023-11-02T08:32:00Z) - Of Mice and Pose: 2D Mouse Pose Estimation from Unlabelled Data and
Synthetic Prior [0.7499722271664145]
We propose an approach for estimating 2D mouse body pose from unlabelled images using a synthetically generated empirical pose prior.
We adapt this method to the limb structure of the mouse and generate the empirical prior of 2D poses from a synthetic 3D mouse model.
In experiments on a new mouse video dataset, we evaluate the performance of the approach by comparing pose predictions to a manually obtained ground truth.
arXiv Detail & Related papers (2023-07-25T09:31:55Z) - Prior-Aware Synthetic Data to the Rescue: Animal Pose Estimation with
Very Limited Real Data [18.06492246414256]
We present a data efficient strategy for pose estimation in quadrupeds that requires only a small amount of real images from the target animal.
It is confirmed that fine-tuning a backbone network with pretrained weights on generic image datasets such as ImageNet can mitigate the high demand for target animal pose data.
We introduce a prior-aware synthetic animal data generation pipeline called PASyn to augment the animal pose data essential for robust pose estimation.
arXiv Detail & Related papers (2022-08-30T01:17:50Z) - SuperAnimal pretrained pose estimation models for behavioral analysis [42.206265576708255]
Quantification of behavior is critical in applications ranging from neuroscience, veterinary medicine and animal conservation efforts.
We present a series of technical innovations that enable a new method, collectively called SuperAnimal, to develop unified foundation models.
arXiv Detail & Related papers (2022-03-14T18:46:57Z) - Learning Dynamics via Graph Neural Networks for Human Pose Estimation
and Tracking [98.91894395941766]
We propose a novel online approach to learning the pose dynamics, which are independent of pose detections in current fame.
Specifically, we derive this prediction of dynamics through a graph neural network(GNN) that explicitly accounts for both spatial-temporal and visual information.
Experiments on PoseTrack 2017 and PoseTrack 2018 datasets demonstrate that the proposed method achieves results superior to the state of the art on both human pose estimation and tracking tasks.
arXiv Detail & Related papers (2021-06-07T16:36:50Z) - Kinematic-Structure-Preserved Representation for Unsupervised 3D Human
Pose Estimation [58.72192168935338]
Generalizability of human pose estimation models developed using supervision on large-scale in-studio datasets remains questionable.
We propose a novel kinematic-structure-preserved unsupervised 3D pose estimation framework, which is not restrained by any paired or unpaired weak supervisions.
Our proposed model employs three consecutive differentiable transformations named as forward-kinematics, camera-projection and spatial-map transformation.
arXiv Detail & Related papers (2020-06-24T23:56:33Z) - Transferring Dense Pose to Proximal Animal Classes [83.84439508978126]
We show that it is possible to transfer the knowledge existing in dense pose recognition for humans, as well as in more general object detectors and segmenters, to the problem of dense pose recognition in other classes.
We do this by establishing a DensePose model for the new animal which is also geometrically aligned to humans.
We also introduce two benchmark datasets labelled in the manner of DensePose for the class chimpanzee and use them to evaluate our approach.
arXiv Detail & Related papers (2020-02-28T21:43:53Z) - Deformation-aware Unpaired Image Translation for Pose Estimation on
Laboratory Animals [56.65062746564091]
We aim to capture the pose of neuroscience model organisms, without using any manual supervision, to study how neural circuits orchestrate behaviour.
Our key contribution is the explicit and independent modeling of appearance, shape and poses in an unpaired image translation framework.
We demonstrate improved pose estimation accuracy on Drosophila melanogaster (fruit fly), Caenorhabditis elegans (worm) and Danio rerio (zebrafish)
arXiv Detail & Related papers (2020-01-23T15:34:11Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.