Learning Part Segmentation from Synthetic Animals

• URL: http://arxiv.org/abs/2311.18661v1
• Date: Thu, 30 Nov 2023 16:10:04 GMT
• Title: Learning Part Segmentation from Synthetic Animals
• Abstract summary: This paper focuses on learning part segmentation from synthetic animals, leveraging the Skinned Multi-Animal Linear (SMAL) models. We benchmark Syn-to-Real animal part segmentation from SAP to PartImageNet, namely SynRealPart, with existing semantic segmentation domain adaptation methods. Remarkably, our third contribution is to reveal that the learned parts from synthetic tiger and horse are transferable across all quadrupeds in PartImageNet.
• Score: 11.350951919924455
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Semantic part segmentation provides an intricate and interpretable understanding of an object, thereby benefiting numerous downstream tasks. However, the need for exhaustive annotations impedes its usage across diverse object types. This paper focuses on learning part segmentation from synthetic animals, leveraging the Skinned Multi-Animal Linear (SMAL) models to scale up existing synthetic data generated by computer-aided design (CAD) animal models. Compared to CAD models, SMAL models generate data with a wider range of poses observed in real-world scenarios. As a result, our first contribution is to construct a synthetic animal dataset of tigers and horses with more pose diversity, termed Synthetic Animal Parts (SAP). We then benchmark Syn-to-Real animal part segmentation from SAP to PartImageNet, namely SynRealPart, with existing semantic segmentation domain adaptation methods and further improve them as our second contribution. Concretely, we examine three Syn-to-Real adaptation methods but observe relative performance drop due to the innate difference between the two tasks. To address this, we propose a simple yet effective method called Class-Balanced Fourier Data Mixing (CB-FDM). Fourier Data Mixing aligns the spectral amplitudes of synthetic images with real images, thereby making the mixed images have more similar frequency content to real images. We further use Class-Balanced Pseudo-Label Re-Weighting to alleviate the imbalanced class distribution. We demonstrate the efficacy of CB-FDM on SynRealPart over previous methods with significant performance improvements. Remarkably, our third contribution is to reveal that the learned parts from synthetic tiger and horse are transferable across all quadrupeds in PartImageNet, further underscoring the utility and potential applications of animal part segmentation.

Related papers

• AP-CAP: Advancing High-Quality Data Synthesis for Animal Pose Estimation via a Controllable Image Generation Pipeline [26.509823105406088]
  We propose a novel Controllable Image Generation Pipeline for animal pose estimation data, termed AP-CAP. Within this pipeline, we introduce a Multi-Modal Animal Image Generation Model capable of synthesizing images with expected poses. We further propose three innovative strategies: (1) Modality-Fusion-Based Animal Image Synthesis Strategy to integrate multi-source appearance representations, (2) Pose-Adjustment-Based Animal Image Synthesis Strategy to dynamically capture diverse pose variations, and (3) Caption-Enhancement-Based Animal Image Synthesis Strategy to enrich visual semantic understanding.
  arXiv  Detail & Related papers  (2025-04-01T03:28:29Z)
• Learning Vision from Models Rivals Learning Vision from Data [54.43596959598465]
  We introduce SynCLR, a novel approach for learning visual representations exclusively from synthetic images and synthetic captions. We synthesize a large dataset of image captions using LLMs, then use an off-the-shelf text-to-image model to generate multiple images corresponding to each synthetic caption. We perform visual representation learning on these synthetic images via contrastive learning, treating images sharing the same caption as positive pairs.
  arXiv  Detail & Related papers  (2023-12-28T18:59:55Z)
• Microscopy Image Segmentation via Point and Shape Regularized Data Synthesis [9.47802391546853]
  We develop a unified pipeline for microscopy image segmentation using synthetically generated training data. Our framework achieves comparable results to models trained on authentic microscopy images with dense labels.
  arXiv  Detail & Related papers  (2023-08-18T22:00:53Z)
• Prior-Aware Synthetic Data to the Rescue: Animal Pose Estimation with Very Limited Real Data [18.06492246414256]
  We present a data efficient strategy for pose estimation in quadrupeds that requires only a small amount of real images from the target animal. It is confirmed that fine-tuning a backbone network with pretrained weights on generic image datasets such as ImageNet can mitigate the high demand for target animal pose data. We introduce a prior-aware synthetic animal data generation pipeline called PASyn to augment the animal pose data essential for robust pose estimation.
  arXiv  Detail & Related papers  (2022-08-30T01:17:50Z)
• Coarse-to-fine Animal Pose and Shape Estimation [67.39635503744395]
  We propose a coarse-to-fine approach to reconstruct 3D animal mesh from a single image. The coarse estimation stage first estimates the pose, shape and translation parameters of the SMAL model. The estimated meshes are then used as a starting point by a graph convolutional network (GCN) to predict a per-vertex deformation in the refinement stage.
  arXiv  Detail & Related papers  (2021-11-16T01:27:20Z)
• MetaPix: Domain Transfer for Semantic Segmentation by Meta Pixel Weighting [1.9671123873378715]
  We learn a pixel-level weighting of the synthetic data by meta-learning, i.e., the learning of weighting should only be minimizing the loss on the target task. Experiments show that our method with only one single meta module can outperform a complicated combination of an adversarial feature alignment, a reconstruction loss, plus a hierarchical weighting at pixel, region and image levels.
  arXiv  Detail & Related papers  (2021-10-05T01:31:00Z)
• Half-Real Half-Fake Distillation for Class-Incremental Semantic Segmentation [84.1985497426083]
  convolutional neural networks are ill-equipped for incremental learning. New classes are available but the initial training data is not retained. We try to address this issue by "inverting" the trained segmentation network to synthesize input images starting from random noise.
  arXiv  Detail & Related papers  (2021-04-02T03:47:16Z)
• Synthetic Convolutional Features for Improved Semantic Segmentation [139.5772851285601]
  We suggest to generate intermediate convolutional features and propose the first synthesis approach that is catered to such intermediate convolutional features. This allows us to generate new features from label masks and include them successfully into the training procedure. Experimental results and analysis on two challenging datasets Cityscapes and ADE20K show that our generated feature improves performance on segmentation tasks.
  arXiv  Detail & Related papers  (2020-09-18T14:12:50Z)
• Multi-Spectral Image Synthesis for Crop/Weed Segmentation in Precision Farming [3.4788711710826083]
  We propose an alternative solution with respect to the common data augmentation methods, applying it to the problem of crop/weed segmentation in precision farming. We create semi-artificial samples by replacing the most relevant object classes (i.e., crop and weeds) with their synthesized counterparts. In addition to RGB data, we take into account also near-infrared (NIR) information, generating four channel multi-spectral synthetic images.
  arXiv  Detail & Related papers  (2020-09-12T08:49:36Z)
• Pairwise Relation Learning for Semi-supervised Gland Segmentation [90.45303394358493]
  We propose a pairwise relation-based semi-supervised (PRS2) model for gland segmentation on histology images. This model consists of a segmentation network (S-Net) and a pairwise relation network (PR-Net) We evaluate our model against five recent methods on the GlaS dataset and three recent methods on the CRAG dataset.
  arXiv  Detail & Related papers  (2020-08-06T15:02:38Z)
• Monocular Human Pose and Shape Reconstruction using Part Differentiable Rendering [53.16864661460889]
  Recent works succeed in regression-based methods which estimate parametric models directly through a deep neural network supervised by 3D ground truth. In this paper, we introduce body segmentation as critical supervision. To improve the reconstruction with part segmentation, we propose a part-level differentiable part that enables part-based models to be supervised by part segmentation.
  arXiv  Detail & Related papers  (2020-03-24T14:25:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.