Comparing Self-Supervised Learning Models Pre-Trained on Human Speech and Animal Vocalizations for Bioacoustics Processing

• URL: http://arxiv.org/abs/2501.05987v2
• Date: Sun, 19 Jan 2025 15:53:31 GMT
• Title: Comparing Self-Supervised Learning Models Pre-Trained on Human Speech and Animal Vocalizations for Bioacoustics Processing
• Authors: Eklavya Sarkar, Mathew Magimai. -Doss,
• Abstract summary: Self-supervised learning (SSL) foundation models have emerged as powerful, domain-agnostic, general-purpose feature extractors. This paper investigates whether SSL models pre-trained directly on animal vocalizations offer a significant advantage over those pre-trained on speech.
• Score: 19.205671029694074
• License:
• Abstract: Self-supervised learning (SSL) foundation models have emerged as powerful, domain-agnostic, general-purpose feature extractors applicable to a wide range of tasks. Such models pre-trained on human speech have demonstrated high transferability for bioacoustic processing. This paper investigates (i) whether SSL models pre-trained directly on animal vocalizations offer a significant advantage over those pre-trained on speech, and (ii) whether fine-tuning speech-pretrained models on automatic speech recognition (ASR) tasks can enhance bioacoustic classification. We conduct a comparative analysis using three diverse bioacoustic datasets and two different bioacoustic tasks. Results indicate that pre-training on bioacoustic data provides only marginal improvements over speech-pretrained models, with comparable performance in most scenarios. Fine-tuning on ASR tasks yields mixed outcomes, suggesting that the general-purpose representations learned during SSL pre-training are already well-suited for bioacoustic tasks. These findings highlight the robustness of speech-pretrained SSL models for bioacoustics and imply that extensive fine-tuning may not be necessary for optimal performance.

Related papers

• On the Utility of Speech and Audio Foundation Models for Marmoset Call Analysis [19.205671029694074]
  This study assesses feature representations derived from speech and general audio domains, across pre-training bandwidths of 4, 8, and 16 kHz for marmoset call-type and caller classification tasks. Results show that models with higher bandwidth improve performance, and pre-training on speech or general audio yields comparable results, improving over a spectral baseline.
  arXiv  Detail & Related papers  (2024-07-23T12:00:44Z)
• Self-supervised ASR Models and Features For Dysarthric and Elderly Speech Recognition [71.87998918300806]
  This paper explores approaches to integrate domain fine-tuned SSL pre-trained models and their features into TDNN and Conformer ASR systems. TDNN systems constructed by integrating domain-adapted HuBERT, wav2vec2-conformer or multi-lingual XLSR models consistently outperform standalone fine-tuned SSL pre-trained models. Consistent improvements in Alzheimer's Disease detection accuracy are also obtained using the DementiaBank Pitt elderly speech recognition outputs.
  arXiv  Detail & Related papers  (2024-07-03T08:33:39Z)
• End-to-End Speech Recognition and Disfluency Removal with Acoustic Language Model Pretraining [0.0]
  We revisit the performance comparison between two-stage and end-to-end model. We find that audio based language models pretrained using weak self-supervised objectives match or exceed the performance of similarly trained two-stage models.
  arXiv  Detail & Related papers  (2023-09-08T17:12:14Z)
• Can Self-Supervised Neural Representations Pre-Trained on Human Speech distinguish Animal Callers? [23.041173892976325]
  Self-supervised learning (SSL) models use only the intrinsic structure of a given signal, independent of its acoustic domain, to extract essential information from the input to an embedding space. This paper explores the cross-transferability of SSL neural representations learned from human speech to analyze bio-acoustic signals.
  arXiv  Detail & Related papers  (2023-05-23T13:06:14Z)
• Analysing the Impact of Audio Quality on the Use of Naturalistic Long-Form Recordings for Infant-Directed Speech Research [62.997667081978825]
  Modelling of early language acquisition aims to understand how infants bootstrap their language skills. Recent developments have enabled the use of more naturalistic training data for computational models. It is currently unclear how the sound quality could affect analyses and modelling experiments conducted on such data.
  arXiv  Detail & Related papers  (2023-05-03T08:25:37Z)
• BEATs: Audio Pre-Training with Acoustic Tokenizers [77.8510930885778]
  Self-supervised learning (SSL) has been witnessed in language, vision, speech, and audio domains over the past few years. We propose BEATs, an iterative audio pre-training framework to learn Bidirectional representation from Audio Transformers. In the first iteration, we use random projection as the acoustic tokenizer to train an audio SSL model in a mask and label prediction manner. Then, we train an acoustic tokenizer for the next iteration by distilling the semantic knowledge from the pre-trained or fine-tuned audio SSL model.
  arXiv  Detail & Related papers  (2022-12-18T10:41:55Z)
• Supervision-Guided Codebooks for Masked Prediction in Speech Pre-training [102.14558233502514]
  Masked prediction pre-training has seen remarkable progress in self-supervised learning (SSL) for speech recognition. We propose two supervision-guided codebook generation approaches to improve automatic speech recognition (ASR) performance.
  arXiv  Detail & Related papers  (2022-06-21T06:08:30Z)
• Self-supervised models of audio effectively explain human cortical responses to speech [71.57870452667369]
  We capitalize on the progress of self-supervised speech representation learning to create new state-of-the-art models of the human auditory system. We show that these results show that self-supervised models effectively capture the hierarchy of information relevant to different stages of speech processing in human cortex.
  arXiv  Detail & Related papers  (2022-05-27T22:04:02Z)
• Deploying self-supervised learning in the wild for hybrid automatic speech recognition [20.03807843795386]
  Self-supervised learning (SSL) methods have proven to be very successful in automatic speech recognition (ASR) We show how to utilize untranscribed audio data in SSL from data pre-processing to deploying an streaming hybrid ASR model.
  arXiv  Detail & Related papers  (2022-05-17T19:37:40Z)
• An Exploration of Prompt Tuning on Generative Spoken Language Model for Speech Processing Tasks [112.1942546460814]
  We report the first exploration of the prompt tuning paradigm for speech processing tasks based on Generative Spoken Language Model (GSLM) Experiment results show that the prompt tuning technique achieves competitive performance in speech classification tasks with fewer trainable parameters than fine-tuning specialized downstream models.
  arXiv  Detail & Related papers  (2022-03-31T03:26:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.