Learning neural audio features without supervision

• URL: http://arxiv.org/abs/2203.15519v1
• Date: Tue, 29 Mar 2022 12:59:08 GMT
• Title: Learning neural audio features without supervision
• Abstract summary: We study the feasibility of combining both approaches, i.e., pre-training learnable jointly with the main architecture for downstream classification. We uncover crucial differences in properties of learned components when used in a supervised and self-supervised setting.
• Score: 14.526173151915662
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deep audio classification, traditionally cast as training a deep neural network on top of mel-filterbanks in a supervised fashion, has recently benefited from two independent lines of work. The first one explores "learnable frontends", i.e., neural modules that produce a learnable time-frequency representation, to overcome limitations of fixed features. The second one uses self-supervised learning to leverage unprecedented scales of pre-training data. In this work, we study the feasibility of combining both approaches, i.e., pre-training learnable frontend jointly with the main architecture for downstream classification. First, we show that pretraining two previously proposed frontends (SincNet and LEAF) on Audioset drastically improves linear-probe performance over fixed mel-filterbanks, suggesting that learnable time-frequency representations can benefit self-supervised pre-training even more than supervised training. Surprisingly, randomly initialized learnable filterbanks outperform mel-scaled initialization in the self-supervised setting, a counter-intuitive result that questions the appropriateness of strong priors when designing learnable filters. Through exploratory analysis of the learned frontend components, we uncover crucial differences in properties of these frontends when used in a supervised and self-supervised setting, especially the affinity of self-supervised filters to diverge significantly from the mel-scale to model a broader range of frequencies.

Related papers

• Masked Autoencoders with Limited Data: Does It Work? A Fine-Grained Bioacoustics Case Study [20.469464200788583]
  Masked autoencoders (MAE) have shown strong transferability on massive audio corpora.<n>We conduct a systematic study of MAE pretraining for species classification on iNatSounds.<n>Our results indicate that, in moderate-sized fine-grained bioacoustic settings, pretraining scale dominates objective design.
  arXiv  Detail & Related papers  (2026-05-13T18:45:08Z)
• Balancing Stability and Plasticity in Sequentially Trained Early-Exiting Neural Networks [11.704570155438887]
  Early-exiting neural networks enable adaptive inference by allowing inputs to exit at intermediate classifiers.<n>We propose two alternative approaches that operate at different levels of the model.<n> Experiments on standard benchmarks show that our approaches consistently improve early-exit performance.
  arXiv  Detail & Related papers  (2026-05-06T18:36:32Z)
• A Theory of How Pretraining Shapes Inductive Bias in Fine-Tuning [51.505728136705564]
  We develop an analytical theory of the pretraining-fine-tuning pipeline in diagonal linear networks.<n>We find that different initialization choices place the network into four distinct fine-tuning regimes.<n>A smaller scale in earlier layers enables the network to both reuse and refine its features, leading to superior generalization.
  arXiv  Detail & Related papers  (2026-02-23T17:19:33Z)
• Self-Supervised Learning via Flow-Guided Neural Operator on Time-Series Data [57.85958428020496]
  Flow-Guided Neural Operator (FGNO) is a novel framework combining operator learning with flow matching for SSL training.<n>FGNO learns mappings in functional spaces by using Short-Time Fourier Transform to unify different time resolutions.<n>Unlike prior generative SSL methods that use noisy inputs during inference, we propose using clean inputs for representation extraction while learning representations with noise.
  arXiv  Detail & Related papers  (2026-02-12T18:54:57Z)
• PredNext: Explicit Cross-View Temporal Prediction for Unsupervised Learning in Spiking Neural Networks [70.1286354746363]
  Spiking Neural Networks (SNNs) offer a natural platform for unsupervised representation learning.<n>Current unsupervised SNNs employ shallow architectures or localized plasticity rules, limiting their ability to model long-range temporal dependencies.<n>We propose PredNext, which explicitly models temporal relationships through cross-view future Step Prediction and Clip Prediction.
  arXiv  Detail & Related papers  (2025-09-29T14:27:58Z)
• Unrolled denoising networks provably learn optimal Bayesian inference [54.79172096306631]
  We prove the first rigorous learning guarantees for neural networks based on unrolling approximate message passing (AMP) For compressed sensing, we prove that when trained on data drawn from a product prior, the layers of the network converge to the same denoisers used in Bayes AMP.
  arXiv  Detail & Related papers  (2024-09-19T17:56:16Z)
• Frequency-Guided Masking for Enhanced Vision Self-Supervised Learning [49.275450836604726]
  We present a novel frequency-based Self-Supervised Learning (SSL) approach that significantly enhances its efficacy for pre-training. We employ a two-branch framework empowered by knowledge distillation, enabling the model to take both the filtered and original images as input.
  arXiv  Detail & Related papers  (2024-09-16T15:10:07Z)
• Learning with Noisy Foundation Models [95.50968225050012]
  This paper is the first work to comprehensively understand and analyze the nature of noise in pre-training datasets. We propose a tuning method (NMTune) to affine the feature space to mitigate the malignant effect of noise and improve generalization.
  arXiv  Detail & Related papers  (2024-03-11T16:22:41Z)
• Battle of the Backbones: A Large-Scale Comparison of Pretrained Models across Computer Vision Tasks [139.3768582233067]
  Battle of the Backbones (BoB) is a benchmarking tool for neural network based computer vision systems. We find that vision transformers (ViTs) and self-supervised learning (SSL) are increasingly popular. In apples-to-apples comparisons on the same architectures and similarly sized pretraining datasets, we find that SSL backbones are highly competitive.
  arXiv  Detail & Related papers  (2023-10-30T18:23:58Z)
• Audio-Visual Efficient Conformer for Robust Speech Recognition [91.3755431537592]
  We propose to improve the noise of the recently proposed Efficient Conformer Connectionist Temporal Classification architecture by processing both audio and visual modalities. Our experiments show that using audio and visual modalities allows to better recognize speech in the presence of environmental noise and significantly accelerate training, reaching lower WER with 4 times less training steps.
  arXiv  Detail & Related papers  (2023-01-04T05:36:56Z)
• Self-Supervised Training Enhances Online Continual Learning [37.91734641808391]
  In continual learning, a system must incrementally learn from a non-stationary data stream without catastrophic forgetting. Self-supervised pre-training could yield features that generalize better than supervised learning. Our best system achieves a 14.95% relative increase in top-1 accuracy on class incremental ImageNet over the prior state of the art for online continual learning.
  arXiv  Detail & Related papers  (2021-03-25T17:45:27Z)
• End-to-end spoken language understanding using transformer networks and self-supervised pre-trained features [17.407912171579852]
  Transformer networks and self-supervised pre-training have consistently delivered state-of-art results in the field of natural language processing (NLP) We introduce a modular End-to-End (E2E) SLU transformer network based architecture which allows the use of self-supervised pre-trained acoustic features.
  arXiv  Detail & Related papers  (2020-11-16T19:30:52Z)
• Fast accuracy estimation of deep learning based multi-class musical source separation [79.10962538141445]
  We propose a method to evaluate the separability of instruments in any dataset without training and tuning a neural network. Based on the oracle principle with an ideal ratio mask, our approach is an excellent proxy to estimate the separation performances of state-of-the-art deep learning approaches.
  arXiv  Detail & Related papers  (2020-10-19T13:05:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.