Masked Autoencoders with Limited Data: Does It Work? A Fine-Grained Bioacoustics Case Study

• URL: http://arxiv.org/abs/2605.14031v1
• Date: Wed, 13 May 2026 18:45:08 GMT
• Title: Masked Autoencoders with Limited Data: Does It Work? A Fine-Grained Bioacoustics Case Study
• Abstract summary: Masked autoencoders (MAE) have shown strong transferability on massive audio corpora.<n>We conduct a systematic study of MAE pretraining for species classification on iNatSounds.<n>Our results indicate that, in moderate-sized fine-grained bioacoustic settings, pretraining scale dominates objective design.
• Score: 20.469464200788583
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Bioacoustic recognition requires fine-grained acoustic understanding to distinguish similar-sounding species. However, many large-scale data repositories such as iNaturalist are weakly annotated, often with only a single positive species label per recording, making supervised learning particularly challenging. Inspired by advances in computer vision, recent approaches have shifted toward self-supervised learning to capture the underlying structure of audio without relying on exhaustive annotations. In particular, masked autoencoders (MAE) have shown strong transferability on massive audio corpora, yet their effectiveness in more modest bioacoustic settings remains underexplored. In this work, we conduct a systematic study of MAE pretraining for species classification on iNatSounds, analyzing the impacts of pretraining data scale, domain specificity, data curation, and transfer strategies. Consistent with prior work, we find that models pretrained on diverse general audio data achieve the best transfer performance on iNatSounds. Contrary to observations from large-scale audio benchmarks, we find that (1) additional masked reconstruction pretraining on domain-specific data provides limited benefits and may even degrade performance relative to off-the-shelf models, and (2) selective data filtering offers a negligible advantage when the overall data scale is limited. Our results indicate that, in moderate-sized fine-grained bioacoustic settings, pretraining scale dominates objective design. These findings further clarify when MAE-based pretraining is effective and provide practical guidance for model selection under limited supervision.

Related papers

• How Class Ontology and Data Scale Affect Audio Transfer Learning [86.30979869026778]
  Transfer learning is a crucial concept within deep learning that allows artificial neural networks to benefit from a large pre-training data basis.<n>We study audio-to-audio transfer learning, in which we pre-train various model states on subsets of AudioSet.<n>We report that increasing the number of samples and classes in the pre-training data both have a positive impact on transfer learning.
  arXiv  Detail & Related papers  (2026-03-26T14:18:29Z)
• Decodable but not structured: linear probing enables Underwater Acoustic Target Recognition with pretrained audio embeddings [1.25052154879199]
  anthropogenic noise from ships contribute significantly to underwater sound pollution, posing risks to marine ecosystems.<n> Passive Acoustic Monitoring (PAM) systems are widely deployed for this purpose, generating years of underwater recordings across diverse soundscapes.<n>Recent advances in automatic Underwater Acoustic Target Recognition (UATR) have largely relied on supervised learning, which is constrained by the scarcity of labeled data.<n>In this work, we conduct the first empirical comparative study of transfer learning for UATR, evaluating multiple pretrained audio models originating from diverse audio domains.
  arXiv  Detail & Related papers  (2026-01-13T09:15:31Z)
• Learning Robust Spatial Representations from Binaural Audio through Feature Distillation [64.36563387033921]
  We investigate the use of a pretraining stage based on feature distillation to learn a robust spatial representation of speech without the need for data labels.<n>Our experiments demonstrate that the pretrained models show improved performance in noisy and reverberant environments.
  arXiv  Detail & Related papers  (2025-08-28T15:43:15Z)
• What Matters for Bioacoustic Encoding [34.118070876417065]
  We present a large-scale empirical study that covers aspects of bioacoustics that are relevant to research.<n>We obtain encoders that are state-of-the-art on the existing and proposed benchmarks.<n>Specifically, across 26 datasets with tasks including species classification, detection, individual ID, and vocal repertoire discovery, we find self-supervised pre-training followed by supervised post-training.
  arXiv  Detail & Related papers  (2025-08-15T23:52:34Z)
• Self-supervised Learning for Acoustic Few-Shot Classification [10.180992026994739]
  We introduce and evaluate a new architecture that combines CNN-based preprocessing with feature extraction based on state space models (SSMs)<n>We pre-train this architecture using contrastive learning on the actual task data and subsequent fine-tuning with an extremely small amount of labelled data.<n>Our evaluation shows that it outperforms state-of-the-art architectures on the few-shot classification problem.
  arXiv  Detail & Related papers  (2024-09-15T07:45:11Z)
• Impact of Noisy Supervision in Foundation Model Learning [91.56591923244943]
  This paper is the first work to comprehensively understand and analyze the nature of noise in pre-training datasets.<n>We propose a tuning method (NMTune) to affine the feature space to mitigate the malignant effect of noise and improve generalization.
  arXiv  Detail & Related papers  (2024-03-11T16:22:41Z)
• WhaleNet: a Novel Deep Learning Architecture for Marine Mammals Vocalizations on Watkins Marine Mammal Sound Database [49.1574468325115]
  We introduce textbfWhaleNet (Wavelet Highly Adaptive Learning Ensemble Network), a sophisticated deep ensemble architecture for the classification of marine mammal vocalizations. We achieve an improvement in classification accuracy by $8-10%$ over existing architectures, corresponding to a classification accuracy of $97.61%$.
  arXiv  Detail & Related papers  (2024-02-20T11:36:23Z)
• Fine tuning Pre trained Models for Robustness Under Noisy Labels [34.68018860186995]
  The presence of noisy labels in a training dataset can significantly impact the performance of machine learning models. We introduce a novel algorithm called TURN, which robustly and efficiently transfers the prior knowledge of pre-trained models.
  arXiv  Detail & Related papers  (2023-10-24T20:28:59Z)
• Understanding and Mitigating the Label Noise in Pre-training on Downstream Tasks [91.15120211190519]
  This paper aims to understand the nature of noise in pre-training datasets and to mitigate its impact on downstream tasks. We propose a light-weight black-box tuning method (NMTune) to affine the feature space to mitigate the malignant effect of noise.
  arXiv  Detail & Related papers  (2023-09-29T06:18:15Z)
• Unsupervised Domain Adaptation for Acoustic Scene Classification Using Band-Wise Statistics Matching [69.24460241328521]
  Machine learning algorithms can be negatively affected by mismatches between training (source) and test (target) data distributions. We propose an unsupervised domain adaptation method that consists of aligning the first- and second-order sample statistics of each frequency band of target-domain acoustic scenes to the ones of the source-domain training dataset. We show that the proposed method outperforms the state-of-the-art unsupervised methods found in the literature in terms of both source- and target-domain classification accuracy.
  arXiv  Detail & Related papers  (2020-04-30T23:56:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.