SPAC-Net: Synthetic Pose-aware Animal ControlNet for Enhanced Pose
Estimation
- URL: http://arxiv.org/abs/2305.17845v2
- Date: Wed, 31 May 2023 11:09:40 GMT
- Title: SPAC-Net: Synthetic Pose-aware Animal ControlNet for Enhanced Pose
Estimation
- Authors: Le Jiang and Sarah Ostadabbas
- Abstract summary: We present a new approach called Synthetic Pose-aware Animal ControlNet (SPAC-Net)
We leverage the plausible pose data generated by the Variational Auto-Encoder (VAE)-based data generation pipeline to generate synthetic data with pose labels closer to real data.
In addition, we propose the Bi-ControlNet structure to separately detect the HED boundary of animals and backgrounds, improving the precision and stability of the generated data.
- Score: 19.035988285379116
- License: http://creativecommons.org/licenses/by-nc-nd/4.0/
- Abstract: Animal pose estimation has become a crucial area of research, but the
scarcity of annotated data is a significant challenge in developing accurate
models. Synthetic data has emerged as a promising alternative, but it
frequently exhibits domain discrepancies with real data. Style transfer
algorithms have been proposed to address this issue, but they suffer from
insufficient spatial correspondence, leading to the loss of label information.
In this work, we present a new approach called Synthetic Pose-aware Animal
ControlNet (SPAC-Net), which incorporates ControlNet into the previously
proposed Prior-Aware Synthetic animal data generation (PASyn) pipeline. We
leverage the plausible pose data generated by the Variational Auto-Encoder
(VAE)-based data generation pipeline as input for the ControlNet
Holistically-nested Edge Detection (HED) boundary task model to generate
synthetic data with pose labels that are closer to real data, making it
possible to train a high-precision pose estimation network without the need for
real data. In addition, we propose the Bi-ControlNet structure to separately
detect the HED boundary of animals and backgrounds, improving the precision and
stability of the generated data. Using the SPAC-Net pipeline, we generate
synthetic zebra and rhino images and test them on the AP10K real dataset,
demonstrating superior performance compared to using only real images or
synthetic data generated by other methods. Our work demonstrates the potential
for synthetic data to overcome the challenge of limited annotated data in
animal pose estimation.
Related papers
- DODA: Diffusion for Object-detection Domain Adaptation in Agriculture [4.549305421261851]
We propose DODA, a data synthesizer that can generate high-quality object detection data for new domains in agriculture.
Specifically, we improve the controllability of layout-to-image through encoding layout as an image, thereby improving the quality of labels.
arXiv Detail & Related papers (2024-03-27T08:16:33Z) - Reliability in Semantic Segmentation: Can We Use Synthetic Data? [52.5766244206855]
This paper challenges cutting-edge generative models to automatically synthesize data for assessing reliability in semantic segmentation.
By fine-tuning Stable Diffusion, we perform zero-shot generation of synthetic data in OOD domains or inpainted with OOD objects.
We demonstrate a high correlation between the performance on synthetic data and the performance on real OOD data, showing the validity approach.
arXiv Detail & Related papers (2023-12-14T18:56:07Z) - ParGANDA: Making Synthetic Pedestrians A Reality For Object Detection [2.7648976108201815]
We propose to use a Generative Adversarial Network (GAN) to close the gap between the real and synthetic data.
Our approach not only produces visually plausible samples but also does not require any labels of the real domain.
arXiv Detail & Related papers (2023-07-21T05:26:32Z) - CasTGAN: Cascaded Generative Adversarial Network for Realistic Tabular
Data Synthesis [0.4999814847776097]
Generative adversarial networks (GANs) have drawn considerable attention in recent years for their proven capability in generating synthetic data.
The validity of the synthetic data and the underlying privacy concerns represent major challenges which are not sufficiently addressed.
arXiv Detail & Related papers (2023-07-01T16:52:18Z) - Exploring the Effectiveness of Dataset Synthesis: An application of
Apple Detection in Orchards [68.95806641664713]
We explore the usability of Stable Diffusion 2.1-base for generating synthetic datasets of apple trees for object detection.
We train a YOLOv5m object detection model to predict apples in a real-world apple detection dataset.
Results demonstrate that the model trained on generated data is slightly underperforming compared to a baseline model trained on real-world images.
arXiv Detail & Related papers (2023-06-20T09:46:01Z) - Synthetic data, real errors: how (not) to publish and use synthetic data [86.65594304109567]
We show how the generative process affects the downstream ML task.
We introduce Deep Generative Ensemble (DGE) to approximate the posterior distribution over the generative process model parameters.
arXiv Detail & Related papers (2023-05-16T07:30:29Z) - A New Benchmark: On the Utility of Synthetic Data with Blender for Bare
Supervised Learning and Downstream Domain Adaptation [42.2398858786125]
Deep learning in computer vision has achieved great success with the price of large-scale labeled training data.
The uncontrollable data collection process produces non-IID training and test data, where undesired duplication may exist.
To circumvent them, an alternative is to generate synthetic data via 3D rendering with domain randomization.
arXiv Detail & Related papers (2023-03-16T09:03:52Z) - Self-Supervised Pre-Training for Transformer-Based Person
Re-Identification [54.55281692768765]
Transformer-based supervised pre-training achieves great performance in person re-identification (ReID)
Due to the domain gap between ImageNet and ReID datasets, it usually needs a larger pre-training dataset to boost the performance.
This work aims to mitigate the gap between the pre-training and ReID datasets from the perspective of data and model structure.
arXiv Detail & Related papers (2021-11-23T18:59:08Z) - Unsupervised Domain Adaptive Learning via Synthetic Data for Person
Re-identification [101.1886788396803]
Person re-identification (re-ID) has gained more and more attention due to its widespread applications in video surveillance.
Unfortunately, the mainstream deep learning methods still need a large quantity of labeled data to train models.
In this paper, we develop a data collector to automatically generate synthetic re-ID samples in a computer game, and construct a data labeler to simultaneously annotate them.
arXiv Detail & Related papers (2021-09-12T15:51:41Z) - Negative Data Augmentation [127.28042046152954]
We show that negative data augmentation samples provide information on the support of the data distribution.
We introduce a new GAN training objective where we use NDA as an additional source of synthetic data for the discriminator.
Empirically, models trained with our method achieve improved conditional/unconditional image generation along with improved anomaly detection capabilities.
arXiv Detail & Related papers (2021-02-09T20:28:35Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.