Unsupervised Discriminative Learning of Sounds for Audio Event Classification

• URL: http://arxiv.org/abs/2105.09279v2
• Date: Thu, 20 May 2021 10:51:57 GMT
• Title: Unsupervised Discriminative Learning of Sounds for Audio Event Classification
• Authors: Sascha Hornauer, Ke Li, Stella X. Yu, Shabnam Ghaffarzadegan, Liu Ren
• Abstract summary: Network-based audio event classification has shown the benefit of pre-training models on visual data such as ImageNet. We show a fast and effective alternative that pre-trains the model unsupervised, only on audio data and yet delivers on-par performance with ImageNet pre-training.
• Score: 43.81789898864507
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Recent progress in network-based audio event classification has shown the benefit of pre-training models on visual data such as ImageNet. While this process allows knowledge transfer across different domains, training a model on large-scale visual datasets is time consuming. On several audio event classification benchmarks, we show a fast and effective alternative that pre-trains the model unsupervised, only on audio data and yet delivers on-par performance with ImageNet pre-training. Furthermore, we show that our discriminative audio learning can be used to transfer knowledge across audio datasets and optionally include ImageNet pre-training.

Related papers

• Transfer Learning for Passive Sonar Classification using Pre-trained Audio and ImageNet Models [39.85805843651649]
  This study compares pre-trained Audio Neural Networks (PANNs) and ImageNet pre-trained models. It was observed that the ImageNet pre-trained models slightly out-perform pre-trained audio models in passive sonar classification.
  arXiv  Detail & Related papers  (2024-09-20T20:13:45Z)
• CACE-Net: Co-guidance Attention and Contrastive Enhancement for Effective Audio-Visual Event Localization [11.525177542345215]
  We introduce CACE-Net, which differs from most existing methods that solely use audio signals to guide visual information. We propose an audio-visual co-guidance attention mechanism that allows for adaptive bi-directional cross-modal attentional guidance. Experiments on the AVE dataset demonstrate that CACE-Net sets a new benchmark in the audio-visual event localization task.
  arXiv  Detail & Related papers  (2024-08-04T07:48:12Z)
• Audio-Visual Generalized Zero-Shot Learning using Pre-Trained Large Multi-Modal Models [53.48409081555687]
  In this work, we explore such large pre-trained models to obtain features, i.e. CLIP for visual features, and CLAP for audio features. We propose a simple yet effective model that only relies on feed-forward neural networks. Our framework achieves state-of-the-art performance on VGGSound-GZSL, UCF-GZSL, and ActivityNet-GZSL.
  arXiv  Detail & Related papers  (2024-04-09T13:39:37Z)
• Text-to-feature diffusion for audio-visual few-shot learning [59.45164042078649]
  Few-shot learning from video data is a challenging and underexplored, yet much cheaper, setup. We introduce a unified audio-visual few-shot video classification benchmark on three datasets. We show that AV-DIFF obtains state-of-the-art performance on our proposed benchmark for audio-visual few-shot learning.
  arXiv  Detail & Related papers  (2023-09-07T17:30:36Z)
• AVFormer: Injecting Vision into Frozen Speech Models for Zero-Shot AV-ASR [79.21857972093332]
  We present AVFormer, a method for augmenting audio-only models with visual information, at the same time performing lightweight domain adaptation. We show that these can be trained on a small amount of weakly labelled video data with minimum additional training time and parameters. We also introduce a simple curriculum scheme during training which we show is crucial to enable the model to jointly process audio and visual information effectively.
  arXiv  Detail & Related papers  (2023-03-29T07:24:28Z)
• Audiovisual Masked Autoencoders [93.22646144125457]
  We show that we can achieve significant improvements on audiovisual downstream classification tasks. We additionally demonstrate the transferability of our representations, achieving state-of-the-art audiovisual results on Epic Kitchens.
  arXiv  Detail & Related papers  (2022-12-09T17:34:53Z)
• Contrastive Audio-Visual Masked Autoencoder [85.53776628515561]
  Contrastive Audio-Visual Masked Auto-Encoder (CAV-MAE) Our fully self-supervised pretrained CAV-MAE achieves a new SOTA accuracy of 65.9% on VGGSound.
  arXiv  Detail & Related papers  (2022-10-02T07:29:57Z)
• Audio-Visual Scene Classification Using A Transfer Learning Based Joint Optimization Strategy [26.975596225131824]
  We propose a joint training framework, using the acoustic features and raw images directly as inputs for the AVSC task. Specifically, we retrieve the bottom layers of pre-trained image models as visual encoder, and jointly optimize the scene classifier and 1D-CNN based acoustic encoder during training.
  arXiv  Detail & Related papers  (2022-04-25T03:37:02Z)
• SSAST: Self-Supervised Audio Spectrogram Transformer [19.09439093130855]
  We propose to pretrain the Audio Spectrogram Transformer (AST) model with joint discriminative and generative masked spectrogram patch modeling (MSPM) using unlabeled audio. We evaluate our pretrained models on both audio and speech classification tasks including audio event classification, keyword spotting, emotion recognition, and speaker identification. To the best of our knowledge, it is the first patch-based self-supervised learning framework in the audio and speech domain, and also the first self-supervised learning framework for AST.
  arXiv  Detail & Related papers  (2021-10-19T07:58:28Z)
• Audiovisual transfer learning for audio tagging and sound event detection [21.574781022415372]
  We study the merit of transfer learning for two sound recognition problems, i.e., audio tagging and sound event detection. We adapt a baseline system utilizing only spectral acoustic inputs to make use of pretrained auditory and visual features. We perform experiments with these modified models on an audiovisual multi-label data set.
  arXiv  Detail & Related papers  (2021-06-09T21:55:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.