Crossing the Species Divide: Transfer Learning from Speech to Animal Sounds

• URL: http://arxiv.org/abs/2509.04166v1
• Date: Thu, 04 Sep 2025 12:39:05 GMT
• Title: Crossing the Species Divide: Transfer Learning from Speech to Animal Sounds
• Authors: Jules Cauzinille, Marius Miron, Olivier Pietquin, Masato Hagiwara, Ricard Marxer, Arnaud Rey, Benoit Favre,
• Abstract summary: Self-supervised speech models have demonstrated impressive performance in speech processing, but their effectiveness on non-speech data remains underexplored.<n>We show that models such as HuBERT, WavLM, and XEUS can generate rich latent representations of animal sounds across taxa.<n>Results are competitive with fine-tuned bioacoustic pre-trained models and show the impact of noise-robust pre-training setups.
• Score: 24.203596224724848
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Self-supervised speech models have demonstrated impressive performance in speech processing, but their effectiveness on non-speech data remains underexplored. We study the transfer learning capabilities of such models on bioacoustic detection and classification tasks. We show that models such as HuBERT, WavLM, and XEUS can generate rich latent representations of animal sounds across taxa. We analyze the models properties with linear probing on time-averaged representations. We then extend the approach to account for the effect of time-wise information with other downstream architectures. Finally, we study the implication of frequency range and noise on performance. Notably, our results are competitive with fine-tuned bioacoustic pre-trained models and show the impact of noise-robust pre-training setups. These findings highlight the potential of speech-based self-supervised learning as an efficient framework for advancing bioacoustic research.

Related papers

• Learning Robust Spatial Representations from Binaural Audio through Feature Distillation [64.36563387033921]
  We investigate the use of a pretraining stage based on feature distillation to learn a robust spatial representation of speech without the need for data labels.<n>Our experiments demonstrate that the pretrained models show improved performance in noisy and reverberant environments.
  arXiv  Detail & Related papers  (2025-08-28T15:43:15Z)
• Synthetic data enables context-aware bioacoustic sound event detection [18.158806322128527]
  We propose a methodology for training foundation models that enhances their in-context learning capabilities.<n>We generate over 8.8 thousand hours of strongly-labeled audio and train a query-by-example, transformer-based model to perform few-shot bioacoustic sound event detection.<n>We make our trained model available via an API, to provide ecologists and ethologists with a training-free tool for bioacoustic sound event detection.
  arXiv  Detail & Related papers  (2025-03-01T02:03:22Z)
• Comparing Self-Supervised Learning Models Pre-Trained on Human Speech and Animal Vocalizations for Bioacoustics Processing [19.205671029694074]
  Self-supervised learning (SSL) foundation models have emerged as powerful, domain-agnostic, general-purpose feature extractors.<n>This paper investigates whether SSL models pre-trained directly on animal vocalizations offer a significant advantage over those pre-trained on speech.
  arXiv  Detail & Related papers  (2025-01-10T14:18:21Z)
• NatureLM-audio: an Audio-Language Foundation Model for Bioacoustics [35.72581102737726]
  We present NatureLM-audio, the first audio-language foundation model specifically designed for bioacoustics.<n>Our training dataset consists of carefully curated text-audio pairs spanning bioacoustics, speech, and music.<n>We demonstrate successful transfer of learned representations from music and speech to bioacoustics, and our model shows promising generalization to unseen taxa and tasks.
  arXiv  Detail & Related papers  (2024-11-11T18:01:45Z)
• On the Utility of Speech and Audio Foundation Models for Marmoset Call Analysis [19.205671029694074]
  This study assesses feature representations derived from speech and general audio domains, across pre-training bandwidths of 4, 8, and 16 kHz for marmoset call-type and caller classification tasks. Results show that models with higher bandwidth improve performance, and pre-training on speech or general audio yields comparable results, improving over a spectral baseline.
  arXiv  Detail & Related papers  (2024-07-23T12:00:44Z)
• Impact of Noisy Supervision in Foundation Model Learning [91.56591923244943]
  This paper is the first work to comprehensively understand and analyze the nature of noise in pre-training datasets.<n>We propose a tuning method (NMTune) to affine the feature space to mitigate the malignant effect of noise and improve generalization.
  arXiv  Detail & Related papers  (2024-03-11T16:22:41Z)
• Understanding and Mitigating the Label Noise in Pre-training on Downstream Tasks [91.15120211190519]
  This paper aims to understand the nature of noise in pre-training datasets and to mitigate its impact on downstream tasks. We propose a light-weight black-box tuning method (NMTune) to affine the feature space to mitigate the malignant effect of noise.
  arXiv  Detail & Related papers  (2023-09-29T06:18:15Z)
• Analysing the Impact of Audio Quality on the Use of Naturalistic Long-Form Recordings for Infant-Directed Speech Research [62.997667081978825]
  Modelling of early language acquisition aims to understand how infants bootstrap their language skills. Recent developments have enabled the use of more naturalistic training data for computational models. It is currently unclear how the sound quality could affect analyses and modelling experiments conducted on such data.
  arXiv  Detail & Related papers  (2023-05-03T08:25:37Z)
• ASiT: Local-Global Audio Spectrogram vIsion Transformer for Event Classification [42.95038619688867]
  ASiT is a novel self-supervised learning framework that captures local and global contextual information by employing group masked model learning and self-distillation. We evaluate our pretrained models on both audio and speech classification tasks, including audio event classification, keyword spotting, and speaker identification.
  arXiv  Detail & Related papers  (2022-11-23T18:21:09Z)
• Self-supervised models of audio effectively explain human cortical responses to speech [71.57870452667369]
  We capitalize on the progress of self-supervised speech representation learning to create new state-of-the-art models of the human auditory system. We show that these results show that self-supervised models effectively capture the hierarchy of information relevant to different stages of speech processing in human cortex.
  arXiv  Detail & Related papers  (2022-05-27T22:04:02Z)
• Self-Supervised Learning for speech recognition with Intermediate layer supervision [52.93758711230248]
  We propose Intermediate Layer Supervision for Self-Supervised Learning (ILS-SSL) ILS-SSL forces the model to concentrate on content information as much as possible by adding an additional SSL loss on the intermediate layers. Experiments on LibriSpeech test-other set show that our method outperforms HuBERT significantly.
  arXiv  Detail & Related papers  (2021-12-16T10:45:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.