Unlocking Strong Supervision: A Data-Centric Study of General-Purpose Audio Pre-Training Methods

• URL: http://arxiv.org/abs/2603.25767v1
• Date: Thu, 26 Mar 2026 07:18:04 GMT
• Title: Unlocking Strong Supervision: A Data-Centric Study of General-Purpose Audio Pre-Training Methods
• Abstract summary: Current audio pre-training seeks to learn unified representations for broad audio understanding tasks.<n>It remains fragmented and is bottlenecked by its reliance on weak, noisy, and scale-limited labels.<n>We argue that the audio field must first establish its own large-scale, strong supervision framework.
• Score: 29.554633767752193
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Current audio pre-training seeks to learn unified representations for broad audio understanding tasks, but it remains fragmented and is fundamentally bottlenecked by its reliance on weak, noisy, and scale-limited labels. Drawing lessons from vision's foundational pre-training blueprint, we argue that the audio field must first establish its own large-scale, strong supervision framework. We introduce a new data-centric pipeline that leverages a high-fidelity captioner to create SOTA-quality captions and the first Unified Tag System (UTS) that bridges speech, music, and environmental sounds. We then conduct a systematic comparative study of different pre-training objectives on these strong source data. Our experiments suggest that data quality and coverage are the primary drivers of performance, while the choice of objective dictates downstream task specialization.

Related papers

• How Class Ontology and Data Scale Affect Audio Transfer Learning [86.30979869026778]
  Transfer learning is a crucial concept within deep learning that allows artificial neural networks to benefit from a large pre-training data basis.<n>We study audio-to-audio transfer learning, in which we pre-train various model states on subsets of AudioSet.<n>We report that increasing the number of samples and classes in the pre-training data both have a positive impact on transfer learning.
  arXiv  Detail & Related papers  (2026-03-26T14:18:29Z)
• Revisiting Audio-language Pretraining for Learning General-purpose Audio Representation [30.42124709340273]
  We identify three key barriers: limited large-scale audio-text corpora, insufficient caption diversity, and lack of systematic exploration and evaluation.<n>Our results demonstrate that audio-language pretraining yields competitive, transferable representations.<n>These findings establish audio-language pretraining as a viable pathway toward general-purpose audio representations.
  arXiv  Detail & Related papers  (2025-11-20T19:17:35Z)
• Understanding and Mitigating the Label Noise in Pre-training on Downstream Tasks [91.15120211190519]
  This paper aims to understand the nature of noise in pre-training datasets and to mitigate its impact on downstream tasks. We propose a light-weight black-box tuning method (NMTune) to affine the feature space to mitigate the malignant effect of noise.
  arXiv  Detail & Related papers  (2023-09-29T06:18:15Z)
• Language-Guided Audio-Visual Source Separation via Trimodal Consistency [64.0580750128049]
  A key challenge in this task is learning to associate the linguistic description of a sound-emitting object to its visual features and the corresponding components of the audio waveform. We adapt off-the-shelf vision-language foundation models to provide pseudo-target supervision via two novel loss functions. We demonstrate the effectiveness of our self-supervised approach on three audio-visual separation datasets.
  arXiv  Detail & Related papers  (2023-03-28T22:45:40Z)
• Leveraging Pre-trained AudioLDM for Sound Generation: A Benchmark Study [33.10311742703679]
  We make the first attempt to investigate the benefits of pre-training on sound generation with AudioLDM. Our study demonstrates the advantages of the pre-trained AudioLDM, especially in data-scarcity scenarios. We benchmark the sound generation task on various frequently-used datasets.
  arXiv  Detail & Related papers  (2023-03-07T12:49:45Z)
• ASiT: Local-Global Audio Spectrogram vIsion Transformer for Event Classification [42.95038619688867]
  ASiT is a novel self-supervised learning framework that captures local and global contextual information by employing group masked model learning and self-distillation. We evaluate our pretrained models on both audio and speech classification tasks, including audio event classification, keyword spotting, and speaker identification.
  arXiv  Detail & Related papers  (2022-11-23T18:21:09Z)
• SLICER: Learning universal audio representations using low-resource self-supervised pre-training [53.06337011259031]
  We present a new Self-Supervised Learning approach to pre-train encoders on unlabeled audio data. Our primary aim is to learn audio representations that can generalize across a large variety of speech and non-speech tasks.
  arXiv  Detail & Related papers  (2022-11-02T23:45:33Z)
• WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing [102.45426364965887]
  We propose a new pre-trained model, WavLM, to solve full-stack downstream speech tasks. WavLM is built based on the HuBERT framework, with an emphasis on both spoken content modeling and speaker identity preservation. We scale up the training dataset from 60k hours to 94k hours of public audio data, and optimize its training procedure for better representation extraction.
  arXiv  Detail & Related papers  (2021-10-26T17:55:19Z)
• SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Training [33.02912456062474]
  We build a single encoder with the BERT objective on unlabeled text together with the w2v-BERT objective on unlabeled speech. We demonstrate that incorporating both speech and text data during pre-training can significantly improve downstream quality on CoVoST2 speech translation.
  arXiv  Detail & Related papers  (2021-10-20T00:59:36Z)
• SSAST: Self-Supervised Audio Spectrogram Transformer [19.09439093130855]
  We propose to pretrain the Audio Spectrogram Transformer (AST) model with joint discriminative and generative masked spectrogram patch modeling (MSPM) using unlabeled audio. We evaluate our pretrained models on both audio and speech classification tasks including audio event classification, keyword spotting, emotion recognition, and speaker identification. To the best of our knowledge, it is the first patch-based self-supervised learning framework in the audio and speech domain, and also the first self-supervised learning framework for AST.
  arXiv  Detail & Related papers  (2021-10-19T07:58:28Z)
• Deep Clustering For General-Purpose Audio Representations [2.8086459907382224]
  We introduce DECAR, a self-supervised pre-training approach for learning general-purpose audio representations. We pre-train DECAR embeddings on a balanced subset of the large-scale Audioset dataset. We transfer those representations to 9 downstream classification tasks, including speech, music, animal sounds, and acoustic scenes.
  arXiv  Detail & Related papers  (2021-10-17T19:03:51Z)
• Self-supervised Text-independent Speaker Verification using Prototypical Momentum Contrastive Learning [58.14807331265752]
  We show that better speaker embeddings can be learned by momentum contrastive learning. We generalize the self-supervised framework to a semi-supervised scenario where only a small portion of the data is labeled.
  arXiv  Detail & Related papers  (2020-12-13T23:23:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.