AniMer+: Unified Pose and Shape Estimation Across Mammalia and Aves via Family-Aware Transformer

• URL: http://arxiv.org/abs/2508.00298v1
• Date: Fri, 01 Aug 2025 03:53:03 GMT
• Title: AniMer+: Unified Pose and Shape Estimation Across Mammalia and Aves via Family-Aware Transformer
• Authors: Jin Lyu, Liang An, Li Lin, Pujin Cheng, Yebin Liu, Xiaoying Tang,
• Abstract summary: We introduce AniMer+, an extended version of our scalable AniMer framework.<n>A key innovation of AniMer+ is its high-capacity, family-aware Vision Transformer (ViT)<n>We produce two large-scale synthetic datasets: CtrlAni3D for quadrupeds and CtrlAVES3D for birds.
• Score: 26.738709781346678
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In the era of foundation models, achieving a unified understanding of different dynamic objects through a single network has the potential to empower stronger spatial intelligence. Moreover, accurate estimation of animal pose and shape across diverse species is essential for quantitative analysis in biological research. However, this topic remains underexplored due to the limited network capacity of previous methods and the scarcity of comprehensive multi-species datasets. To address these limitations, we introduce AniMer+, an extended version of our scalable AniMer framework. In this paper, we focus on a unified approach for reconstructing mammals (mammalia) and birds (aves). A key innovation of AniMer+ is its high-capacity, family-aware Vision Transformer (ViT) incorporating a Mixture-of-Experts (MoE) design. Its architecture partitions network layers into taxa-specific components (for mammalia and aves) and taxa-shared components, enabling efficient learning of both distinct and common anatomical features within a single model. To overcome the critical shortage of 3D training data, especially for birds, we introduce a diffusion-based conditional image generation pipeline. This pipeline produces two large-scale synthetic datasets: CtrlAni3D for quadrupeds and CtrlAVES3D for birds. To note, CtrlAVES3D is the first large-scale, 3D-annotated dataset for birds, which is crucial for resolving single-view depth ambiguities. Trained on an aggregated collection of 41.3k mammalian and 12.4k avian images (combining real and synthetic data), our method demonstrates superior performance over existing approaches across a wide range of benchmarks, including the challenging out-of-domain Animal Kingdom dataset. Ablation studies confirm the effectiveness of both our novel network architecture and the generated synthetic datasets in enhancing real-world application performance.

Related papers

• UAVTwin: Neural Digital Twins for UAVs using Gaussian Splatting [57.63613048492219]
  We present UAVTwin, a method for creating digital twins from real-world environments and facilitating data augmentation for training downstream models embedded in unmanned aerial vehicles (UAVs)<n>This is achieved by integrating 3D Gaussian Splatting (3DGS) for reconstructing backgrounds along with controllable synthetic human models that display diverse appearances and actions in multiple poses.
  arXiv  Detail & Related papers  (2025-04-02T22:17:30Z)
• AP-CAP: Advancing High-Quality Data Synthesis for Animal Pose Estimation via a Controllable Image Generation Pipeline [26.509823105406088]
  We propose a novel Controllable Image Generation Pipeline for animal pose estimation data, termed AP-CAP.<n>Within this pipeline, we introduce a Multi-Modal Animal Image Generation Model capable of synthesizing images with expected poses.<n>We further propose three innovative strategies: (1) Modality-Fusion-Based Animal Image Synthesis Strategy to integrate multi-source appearance representations, (2) Pose-Adjustment-Based Animal Image Synthesis Strategy to dynamically capture diverse pose variations, and (3) Caption-Enhancement-Based Animal Image Synthesis Strategy to enrich visual semantic understanding.
  arXiv  Detail & Related papers  (2025-04-01T03:28:29Z)
• Multispecies Animal Re-ID Using a Large Community-Curated Dataset [0.19418036471925312]
  We construct a dataset that includes 49 species, 37K individual animals, and 225K images, using this data to train a single embedding network for all species.<n>Our model consistently outperforms models trained separately on each species, achieving an average gain of 12.5% in top-1 accuracy.<n>The model is already in production use for 60+ species in a large-scale wildlife monitoring system.
  arXiv  Detail & Related papers  (2024-12-07T09:56:33Z)
• AniMer: Animal Pose and Shape Estimation Using Family Aware Transformer [29.97192007630272]
  This paper presents AniMer to estimate animal pose and shape using family aware Transformer.<n>A key insight of AniMer is its integration of a high-capacity Transformer-based backbone and an animal family supervised contrastive learning scheme.<n>For effective training, we aggregate most available open-sourced quadrupedal datasets, either with 3D or 2D labels.
  arXiv  Detail & Related papers  (2024-12-01T14:59:57Z)
• Graph and Skipped Transformer: Exploiting Spatial and Temporal Modeling Capacities for Efficient 3D Human Pose Estimation [36.93661496405653]
  We take a global approach to exploit Transformer-temporal information with a concise Graph and Skipped Transformer architecture. Specifically, in 3D pose stage, coarse-grained body parts are deployed to construct a fully data-driven adaptive model. Experiments are conducted on Human3.6M, MPI-INF-3DHP and Human-Eva benchmarks.
  arXiv  Detail & Related papers  (2024-07-03T10:42:09Z)
• Pushing Auto-regressive Models for 3D Shape Generation at Capacity and Scalability [118.26563926533517]
  Auto-regressive models have achieved impressive results in 2D image generation by modeling joint distributions in grid space. We extend auto-regressive models to 3D domains, and seek a stronger ability of 3D shape generation by improving auto-regressive models at capacity and scalability simultaneously.
  arXiv  Detail & Related papers  (2024-02-19T15:33:09Z)
• Learning the 3D Fauna of the Web [70.01196719128912]
  We develop 3D-Fauna, an approach that learns a pan-category deformable 3D animal model for more than 100 animal species jointly. One crucial bottleneck of modeling animals is the limited availability of training data. We show that prior category-specific attempts fail to generalize to rare species with limited training images.
  arXiv  Detail & Related papers  (2024-01-04T18:32:48Z)
• The effect of data augmentation and 3D-CNN depth on Alzheimer's Disease detection [51.697248252191265]
  This work summarizes and strictly observes best practices regarding data handling, experimental design, and model evaluation. We focus on Alzheimer's Disease (AD) detection, which serves as a paradigmatic example of challenging problem in healthcare. Within this framework, we train predictive 15 models, considering three different data augmentation strategies and five distinct 3D CNN architectures.
  arXiv  Detail & Related papers  (2023-09-13T10:40:41Z)
• Domain Generalization via Ensemble Stacking for Face Presentation Attack Detection [4.61143637299349]
  Face Presentation Attack Detection (PAD) plays a pivotal role in securing face recognition systems against spoofing attacks. This work proposes a comprehensive solution that combines synthetic data generation and deep ensemble learning. Experimental results on four datasets demonstrate low half total error rates (HTERs) on three benchmark datasets.
  arXiv  Detail & Related papers  (2023-01-05T16:44:36Z)
• Back to MLP: A Simple Baseline for Human Motion Prediction [59.18776744541904]
  This paper tackles the problem of human motion prediction, consisting in forecasting future body poses from historically observed sequences. We show that the performance of these approaches can be surpassed by a light-weight and purely architectural architecture with only 0.14M parameters. An exhaustive evaluation on Human3.6M, AMASS and 3DPW datasets shows that our method, which we dub siMLPe, consistently outperforms all other approaches.
  arXiv  Detail & Related papers  (2022-07-04T16:35:58Z)
• Coarse-to-fine Animal Pose and Shape Estimation [67.39635503744395]
  We propose a coarse-to-fine approach to reconstruct 3D animal mesh from a single image. The coarse estimation stage first estimates the pose, shape and translation parameters of the SMAL model. The estimated meshes are then used as a starting point by a graph convolutional network (GCN) to predict a per-vertex deformation in the refinement stage.
  arXiv  Detail & Related papers  (2021-11-16T01:27:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.