AniMer+: Unified Pose and Shape Estimation Across Mammalia and Aves via Family-Aware Transformer

• URL: http://arxiv.org/abs/2508.00298v1
• Date: Fri, 01 Aug 2025 03:53:03 GMT
• Title: AniMer+: Unified Pose and Shape Estimation Across Mammalia and Aves via Family-Aware Transformer
• Abstract summary: We introduce AniMer+, an extended version of our scalable AniMer framework.<n>A key innovation of AniMer+ is its high-capacity, family-aware Vision Transformer (ViT)<n>We produce two large-scale synthetic datasets: CtrlAni3D for quadrupeds and CtrlAVES3D for birds.
• Score: 26.738709781346678
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In the era of foundation models, achieving a unified understanding of different dynamic objects through a single network has the potential to empower stronger spatial intelligence. Moreover, accurate estimation of animal pose and shape across diverse species is essential for quantitative analysis in biological research. However, this topic remains underexplored due to the limited network capacity of previous methods and the scarcity of comprehensive multi-species datasets. To address these limitations, we introduce AniMer+, an extended version of our scalable AniMer framework. In this paper, we focus on a unified approach for reconstructing mammals (mammalia) and birds (aves). A key innovation of AniMer+ is its high-capacity, family-aware Vision Transformer (ViT) incorporating a Mixture-of-Experts (MoE) design. Its architecture partitions network layers into taxa-specific components (for mammalia and aves) and taxa-shared components, enabling efficient learning of both distinct and common anatomical features within a single model. To overcome the critical shortage of 3D training data, especially for birds, we introduce a diffusion-based conditional image generation pipeline. This pipeline produces two large-scale synthetic datasets: CtrlAni3D for quadrupeds and CtrlAVES3D for birds. To note, CtrlAVES3D is the first large-scale, 3D-annotated dataset for birds, which is crucial for resolving single-view depth ambiguities. Trained on an aggregated collection of 41.3k mammalian and 12.4k avian images (combining real and synthetic data), our method demonstrates superior performance over existing approaches across a wide range of benchmarks, including the challenging out-of-domain Animal Kingdom dataset. Ablation studies confirm the effectiveness of both our novel network architecture and the generated synthetic datasets in enhancing real-world application performance.

Related papers

• PRIMA: Boosting Animal Mesh Recovery with Biological Priors and Test-Time Adaptation [3.599033387924161]
  PRIMA is a framework for robust 3D quadruped mesh recovery under severe species and pose imbalance.<n>We incorporate BioCLIP embeddings as biological priors to inject semantic and morphological knowledge into the reconstruction process.<n>We construct Quadruped3D, a large-scale pseudo-3D dataset that covers diverse species and pose variations.
  arXiv  Detail & Related papers  (2026-06-01T15:13:47Z)
• WildDepth: A Multimodal Dataset for 3D Wildlife Perception and Depth Estimation [22.96845355447165]
  We present WildDepth, a dataset and benchmark suite for depth estimation, behavior detection, and 3D reconstruction from diverse categories of animals.<n>Results show that the use of multi-modal data improves depth reliability by up to 10% RMSE, while RGB-LiDAR fusion enhances 3D reconstruction fidelity by 12% in Chamfer distance.
  arXiv  Detail & Related papers  (2026-03-17T17:19:43Z)
• UniSH: Unifying Scene and Human Reconstruction in a Feed-Forward Pass [83.7071371474926]
  UniSH is a unified, feed-forward framework for joint metric-scale 3D scene and human reconstruction.<n>Our framework bridges strong, disparate priors from scene reconstruction and HMR.<n>Our model achieves state-of-the-art performance on human-centric scene reconstruction.
  arXiv  Detail & Related papers  (2026-01-03T16:06:27Z)
• Particulate: Feed-Forward 3D Object Articulation [89.78788418174946]
  Particulate is a feed-forward approach that, given a single static 3D mesh of an everyday object, directly infers all attributes of the underlying articulated structure.<n>We train the network end-to-end on a diverse collection of articulated 3D assets from public datasets.<n>During inference, Particulate lifts the network's feed-forward prediction to the input mesh, yielding a fully articulated 3D model in seconds.
  arXiv  Detail & Related papers  (2025-12-12T18:59:51Z)
• MORPH: Shape-agnostic PDE Foundation Models [37.26306668589026]
  MORPH is a shape-agnostic, autoregressive foundation model for partial differential equations (PDEs)<n>It is built on a convolutional vision backbone that seamlessly handles heterogeneous evaluations of varying data dimensionality (1D--3D)<n>Across extensive datasets, MORPH matches or surpasses strong baselines and recent state-of-the-art models.
  arXiv  Detail & Related papers  (2025-09-25T22:38:36Z)
• UAVTwin: Neural Digital Twins for UAVs using Gaussian Splatting [57.63613048492219]
  We present UAVTwin, a method for creating digital twins from real-world environments and facilitating data augmentation for training downstream models embedded in unmanned aerial vehicles (UAVs)<n>This is achieved by integrating 3D Gaussian Splatting (3DGS) for reconstructing backgrounds along with controllable synthetic human models that display diverse appearances and actions in multiple poses.
  arXiv  Detail & Related papers  (2025-04-02T22:17:30Z)
• AP-CAP: Advancing High-Quality Data Synthesis for Animal Pose Estimation via a Controllable Image Generation Pipeline [26.509823105406088]
  We propose a novel Controllable Image Generation Pipeline for animal pose estimation data, termed AP-CAP.<n>Within this pipeline, we introduce a Multi-Modal Animal Image Generation Model capable of synthesizing images with expected poses.<n>We further propose three innovative strategies: (1) Modality-Fusion-Based Animal Image Synthesis Strategy to integrate multi-source appearance representations, (2) Pose-Adjustment-Based Animal Image Synthesis Strategy to dynamically capture diverse pose variations, and (3) Caption-Enhancement-Based Animal Image Synthesis Strategy to enrich visual semantic understanding.
  arXiv  Detail & Related papers  (2025-04-01T03:28:29Z)
• Multispecies Animal Re-ID Using a Large Community-Curated Dataset [0.19418036471925312]
  We construct a dataset that includes 49 species, 37K individual animals, and 225K images, using this data to train a single embedding network for all species.<n>Our model consistently outperforms models trained separately on each species, achieving an average gain of 12.5% in top-1 accuracy.<n>The model is already in production use for 60+ species in a large-scale wildlife monitoring system.
  arXiv  Detail & Related papers  (2024-12-07T09:56:33Z)
• AniMer: Animal Pose and Shape Estimation Using Family Aware Transformer [29.97192007630272]
  This paper presents AniMer to estimate animal pose and shape using family aware Transformer.<n>A key insight of AniMer is its integration of a high-capacity Transformer-based backbone and an animal family supervised contrastive learning scheme.<n>For effective training, we aggregate most available open-sourced quadrupedal datasets, either with 3D or 2D labels.
  arXiv  Detail & Related papers  (2024-12-01T14:59:57Z)
• Graph and Skipped Transformer: Exploiting Spatial and Temporal Modeling Capacities for Efficient 3D Human Pose Estimation [36.93661496405653]
  We take a global approach to exploit Transformer-temporal information with a concise Graph and Skipped Transformer architecture. Specifically, in 3D pose stage, coarse-grained body parts are deployed to construct a fully data-driven adaptive model. Experiments are conducted on Human3.6M, MPI-INF-3DHP and Human-Eva benchmarks.
  arXiv  Detail & Related papers  (2024-07-03T10:42:09Z)
• Pushing Auto-regressive Models for 3D Shape Generation at Capacity and Scalability [118.26563926533517]
  Auto-regressive models have achieved impressive results in 2D image generation by modeling joint distributions in grid space. We extend auto-regressive models to 3D domains, and seek a stronger ability of 3D shape generation by improving auto-regressive models at capacity and scalability simultaneously.
  arXiv  Detail & Related papers  (2024-02-19T15:33:09Z)
• Learning the 3D Fauna of the Web [70.01196719128912]
  We develop 3D-Fauna, an approach that learns a pan-category deformable 3D animal model for more than 100 animal species jointly. One crucial bottleneck of modeling animals is the limited availability of training data. We show that prior category-specific attempts fail to generalize to rare species with limited training images.
  arXiv  Detail & Related papers  (2024-01-04T18:32:48Z)
• The effect of data augmentation and 3D-CNN depth on Alzheimer's Disease detection [51.697248252191265]
  This work summarizes and strictly observes best practices regarding data handling, experimental design, and model evaluation. We focus on Alzheimer's Disease (AD) detection, which serves as a paradigmatic example of challenging problem in healthcare. Within this framework, we train predictive 15 models, considering three different data augmentation strategies and five distinct 3D CNN architectures.
  arXiv  Detail & Related papers  (2023-09-13T10:40:41Z)
• Domain Generalization via Ensemble Stacking for Face Presentation Attack Detection [4.61143637299349]
  Face Presentation Attack Detection (PAD) plays a pivotal role in securing face recognition systems against spoofing attacks. This work proposes a comprehensive solution that combines synthetic data generation and deep ensemble learning. Experimental results on four datasets demonstrate low half total error rates (HTERs) on three benchmark datasets.
  arXiv  Detail & Related papers  (2023-01-05T16:44:36Z)
• Back to MLP: A Simple Baseline for Human Motion Prediction [59.18776744541904]
  This paper tackles the problem of human motion prediction, consisting in forecasting future body poses from historically observed sequences. We show that the performance of these approaches can be surpassed by a light-weight and purely architectural architecture with only 0.14M parameters. An exhaustive evaluation on Human3.6M, AMASS and 3DPW datasets shows that our method, which we dub siMLPe, consistently outperforms all other approaches.
  arXiv  Detail & Related papers  (2022-07-04T16:35:58Z)
• Coarse-to-fine Animal Pose and Shape Estimation [67.39635503744395]
  We propose a coarse-to-fine approach to reconstruct 3D animal mesh from a single image. The coarse estimation stage first estimates the pose, shape and translation parameters of the SMAL model. The estimated meshes are then used as a starting point by a graph convolutional network (GCN) to predict a per-vertex deformation in the refinement stage.
  arXiv  Detail & Related papers  (2021-11-16T01:27:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.