Self-supervised Graphs for Audio Representation Learning with Limited Labeled Data

• URL: http://arxiv.org/abs/2202.00097v1
• Date: Mon, 31 Jan 2022 21:32:22 GMT
• Title: Self-supervised Graphs for Audio Representation Learning with Limited Labeled Data
• Authors: Amir Shirian, Krishna Somandepalli, Tanaya Guha
• Abstract summary: Subgraphs are constructed by sampling the entire pool of available training data to exploit the relationship between labelled and unlabeled audio samples. We evaluate our model on three benchmark audio databases, and two tasks: acoustic event detection and speech emotion recognition. Our model is compact (240k parameters), and can produce generalized audio representations that are robust to different types of signal noise.
• Score: 24.608764078208953
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large scale databases with high-quality manual annotations are scarce in audio domain. We thus explore a self-supervised graph approach to learning audio representations from highly limited labelled data. Considering each audio sample as a graph node, we propose a subgraph-based framework with novel self-supervision tasks that can learn effective audio representations. During training, subgraphs are constructed by sampling the entire pool of available training data to exploit the relationship between the labelled and unlabeled audio samples. During inference, we use random edges to alleviate the overhead of graph construction. We evaluate our model on three benchmark audio databases, and two tasks: acoustic event detection and speech emotion recognition. Our semi-supervised model performs better or on par with fully supervised models and outperforms several competitive existing models. Our model is compact (240k parameters), and can produce generalized audio representations that are robust to different types of signal noise.

Related papers

• AV-SUPERB: A Multi-Task Evaluation Benchmark for Audio-Visual Representation Models [92.92233932921741]
  We propose the AV-SUPERB benchmark that enables general-purpose evaluation of unimodal audio/visual and bimodal fusion representations. We evaluate 5 recent self-supervised models and show that none of these models generalize to all tasks. We show that representations may be improved with intermediate-task fine-tuning and audio event classification with AudioSet serves as a strong intermediate task.
  arXiv  Detail & Related papers  (2023-09-19T17:35:16Z)
• Self-Supervised Visual Acoustic Matching [63.492168778869726]
  Acoustic matching aims to re-synthesize an audio clip to sound as if it were recorded in a target acoustic environment. We propose a self-supervised approach to visual acoustic matching where training samples include only the target scene image and audio. Our approach jointly learns to disentangle room acoustics and re-synthesize audio into the target environment, via a conditional GAN framework and a novel metric.
  arXiv  Detail & Related papers  (2023-07-27T17:59:59Z)
• Anomalous Sound Detection using Audio Representation with Machine ID based Contrastive Learning Pretraining [52.191658157204856]
  This paper uses contrastive learning to refine audio representations for each machine ID, rather than for each audio sample. The proposed two-stage method uses contrastive learning to pretrain the audio representation model. Experiments show that our method outperforms the state-of-the-art methods using contrastive learning or self-supervised classification.
  arXiv  Detail & Related papers  (2023-04-07T11:08:31Z)
• Unraveling Instance Associations: A Closer Look for Audio-Visual Segmentation [18.001730255429347]
  Audio-visual segmentation (AVS) is a challenging task that involves accurately segmenting sounding objects based on audio-visual cues. We propose a new cost-effective strategy to build challenging and relatively unbiased high-quality audio-visual segmentation benchmarks. Experiments conducted on existing AVS datasets and on our new benchmark show that our method achieves state-of-the-art (SOTA) segmentation accuracy.
  arXiv  Detail & Related papers  (2023-04-06T09:54:06Z)
• SLICER: Learning universal audio representations using low-resource self-supervised pre-training [53.06337011259031]
  We present a new Self-Supervised Learning approach to pre-train encoders on unlabeled audio data. Our primary aim is to learn audio representations that can generalize across a large variety of speech and non-speech tasks.
  arXiv  Detail & Related papers  (2022-11-02T23:45:33Z)
• A Single Self-Supervised Model for Many Speech Modalities Enables Zero-Shot Modality Transfer [31.028408352051684]
  We present u-HuBERT, a self-supervised pre-training framework that can leverage both multimodal and unimodal speech. Our single model yields 1.2%/1.4%/27.2% speech recognition word error rate on LRS3 with audio-visual/audio/visual input.
  arXiv  Detail & Related papers  (2022-07-14T16:21:33Z)
• Automatic Curation of Large-Scale Datasets for Audio-Visual Representation Learning [62.47593143542552]
  We describe a subset optimization approach for automatic dataset curation. We demonstrate that our approach finds videos with high audio-visual correspondence and show that self-supervised models trained on our data, despite being automatically constructed, achieve similar downstream performances to existing video datasets with similar scales.
  arXiv  Detail & Related papers  (2021-01-26T14:27:47Z)
• A Framework for Generative and Contrastive Learning of Audio Representations [2.8935588665357077]
  We present a framework for contrastive learning for audio representations in a self supervised frame work without access to ground truth labels. We also explore generative models based on state of the art transformer based architectures for learning latent spaces for audio signals. Our system achieves considerable performance, compared to a fully supervised method, with access to ground truth labels to train the neural network model.
  arXiv  Detail & Related papers  (2020-10-22T05:52:32Z)
• COALA: Co-Aligned Autoencoders for Learning Semantically Enriched Audio Representations [32.456824945999465]
  We propose a method for learning audio representations, aligning the learned latent representations of audio and associated tags. We evaluate the quality of our embedding model, measuring its performance as a feature extractor on three different tasks.
  arXiv  Detail & Related papers  (2020-06-15T13:17:18Z)
• High-Fidelity Audio Generation and Representation Learning with Guided Adversarial Autoencoder [2.6770746621108654]
  We propose a new autoencoder based model named "Guided Adversarial Autoencoder (GAAE)" Our proposed model can generate audio with superior quality, which is indistinguishable from the real audio samples.
  arXiv  Detail & Related papers  (2020-06-01T12:19:32Z)
• Audio ALBERT: A Lite BERT for Self-supervised Learning of Audio Representation [51.37980448183019]
  We propose Audio ALBERT, a lite version of the self-supervised speech representation model. We show that Audio ALBERT is capable of achieving competitive performance with those huge models in the downstream tasks. In probing experiments, we find that the latent representations encode richer information of both phoneme and speaker than that of the last layer.
  arXiv  Detail & Related papers  (2020-05-18T10:42:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.