AnimalCLAP: Taxonomy-Aware Language-Audio Pretraining for Species Recognition and Trait Inference

• URL: http://arxiv.org/abs/2603.22053v1
• Date: Mon, 23 Mar 2026 14:53:02 GMT
• Title: AnimalCLAP: Taxonomy-Aware Language-Audio Pretraining for Species Recognition and Trait Inference
• Abstract summary: Animal vocalizations provide crucial insights for wildlife assessment, particularly in complex environments such as forests.<n>Recent advances in deep learning have enabled automatic species classification from their vocalizations.<n>We introduce AnimalCLAP, a taxonomy-aware language-audio framework comprising a new dataset and model that incorporate hierarchical biological information.
• Score: 39.07403450813524
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Animal vocalizations provide crucial insights for wildlife assessment, particularly in complex environments such as forests, aiding species identification and ecological monitoring. Recent advances in deep learning have enabled automatic species classification from their vocalizations. However, classifying species unseen during training remains challenging. To address this limitation, we introduce AnimalCLAP, a taxonomy-aware language-audio framework comprising a new dataset and model that incorporate hierarchical biological information. Specifically, our vocalization dataset consists of 4,225 hours of recordings covering 6,823 species, annotated with 22 ecological traits. The AnimalCLAP model is trained on this dataset to align audio and textual representations using taxonomic structures, improving the recognition of unseen species. We demonstrate that our proposed model effectively infers ecological and biological attributes of species directly from their vocalizations, achieving superior performance compared to CLAP. Our dataset, code, and models will be publicly available at https://dahlian00.github.io/AnimalCLAP_Page/.

Related papers

• CoarseSoundNet: Building a reliable model for ecological soundscape analysis [73.44688723989053]
  A soundscape is composed of three types of sound: biophony (sounds made by animals), geophony (natural abiotic sounds) and anthropophony (sounds made by humans)
  arXiv  Detail & Related papers  (2026-05-20T13:18:11Z)
• The iNaturalist Sounds Dataset [60.157076990024606]
  iNatSounds is a collection of 230,000 audio files capturing sounds from over 5,500 species, contributed by more than 27,000 recordists worldwide.<n>The dataset encompasses sounds from birds, mammals, insects, reptiles, and amphibians, with audio and species labels derived from observations submitted to iNaturalist.<n>We envision models trained on this data powering next-generation public engagement applications, and assisting biologists, ecologists, and land use managers in processing large audio collections.
  arXiv  Detail & Related papers  (2025-05-31T02:07:37Z)
• BioCLIP 2: Emergent Properties from Scaling Hierarchical Contrastive Learning [60.80381372245902]
  We find emergent behaviors in biological vision models via large-scale contrastive vision-language training.<n>We train BioCLIP 2 on TreeOfLife-200M to distinguish different species.<n>We identify emergent properties in the learned embedding space of BioCLIP 2.
  arXiv  Detail & Related papers  (2025-05-29T17:48:20Z)
• NatureLM-audio: an Audio-Language Foundation Model for Bioacoustics [35.72581102737726]
  We present NatureLM-audio, the first audio-language foundation model specifically designed for bioacoustics.<n>Our training dataset consists of carefully curated text-audio pairs spanning bioacoustics, speech, and music.<n>We demonstrate successful transfer of learned representations from music and speech to bioacoustics, and our model shows promising generalization to unseen taxa and tasks.
  arXiv  Detail & Related papers  (2024-11-11T18:01:45Z)
• Generalization in birdsong classification: impact of transfer learning methods and dataset characteristics [2.6740633963478095]
  We explore the effectiveness of transfer learning in large-scale bird sound classification. Our experiments demonstrate that both fine-tuning and knowledge distillation yield strong performance. We advocate for more comprehensive labeling practices within the animal sound community.
  arXiv  Detail & Related papers  (2024-09-21T11:33:12Z)
• Advanced Framework for Animal Sound Classification With Features Optimization [35.2832738406242]
  We propose an automated classification framework applicable to general animal sound classification. Our approach consistently outperforms baseline methods by over 25% in precision, recall, and accuracy.
  arXiv  Detail & Related papers  (2024-07-03T18:33:47Z)
• animal2vec and MeerKAT: A self-supervised transformer for rare-event raw audio input and a large-scale reference dataset for bioacoustics [2.1019401515721583]
  animal2vec is an interpretable large transformer model that learns from unlabeled audio and refines its understanding with labeled data. Meerkat Audio Transcripts is the largest labeled dataset on non-human terrestrial mammals. Our model outperforms existing methods on MeerKAT and the publicly available NIPS4Bplus birdsong dataset.
  arXiv  Detail & Related papers  (2024-06-03T12:11:01Z)
• WhaleNet: a Novel Deep Learning Architecture for Marine Mammals Vocalizations on Watkins Marine Mammal Sound Database [49.1574468325115]
  We introduce textbfWhaleNet (Wavelet Highly Adaptive Learning Ensemble Network), a sophisticated deep ensemble architecture for the classification of marine mammal vocalizations. We achieve an improvement in classification accuracy by $8-10%$ over existing architectures, corresponding to a classification accuracy of $97.61%$.
  arXiv  Detail & Related papers  (2024-02-20T11:36:23Z)
• Transferable Models for Bioacoustics with Human Language Supervision [0.0]
  BioLingual is a new model for bioacoustics based on contrastive language-audio pretraining. It can identify over a thousand species' calls across taxa, complete bioacoustic tasks zero-shot, and retrieve animal vocalization recordings from natural text queries.
  arXiv  Detail & Related papers  (2023-08-09T14:22:18Z)
• Cetacean Translation Initiative: a roadmap to deciphering the communication of sperm whales [97.41394631426678]
  Recent research showed the promise of machine learning tools for analyzing acoustic communication in nonhuman species. We outline the key elements required for the collection and processing of massive bioacoustic data of sperm whales. The technological capabilities developed are likely to yield cross-applications and advancements in broader communities investigating non-human communication and animal behavioral research.
  arXiv  Detail & Related papers  (2021-04-17T18:39:22Z)
• Introducing a Central African Primate Vocalisation Dataset for Automated Species Classification [4.322922553076088]
  We introduce the collected dataset, describe our approach and initial results. We condensed the recordings with an energy/change based automatic vocalisation detection. Initial results reveal up to 82% unweighted average recall (UAR) test set performance in four-class primate species classification.
  arXiv  Detail & Related papers  (2021-01-25T20:21:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.