BYOL for Audio: Self-Supervised Learning for General-Purpose Audio Representation

• URL: http://arxiv.org/abs/2103.06695v1
• Date: Thu, 11 Mar 2021 14:32:33 GMT
• Title: BYOL for Audio: Self-Supervised Learning for General-Purpose Audio Representation
• Authors: Daisuke Niizumi, Daiki Takeuchi, Yasunori Ohishi, Noboru Harada, and Kunio Kashino
• Abstract summary: BYOL-A is an audio self-supervised learning method based on BYOL for learning general-purpose audio representation. With a combination of normalization and augmentation techniques, BYOL-A achieves state-of-the-art results in various downstream tasks.
• Score: 40.116109908079935
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Inspired by the recent progress in self-supervised learning for computer vision that generates supervision using data augmentations, we explore a new general-purpose audio representation learning approach. We propose learning general-purpose audio representation from a single audio segment without expecting relationships between different time segments of audio samples. To implement this principle, we introduce Bootstrap Your Own Latent (BYOL) for Audio (BYOL-A, pronounced "viola"), an audio self-supervised learning method based on BYOL for learning general-purpose audio representation. Unlike most previous audio self-supervised learning methods that rely on agreement of vicinity audio segments or disagreement of remote ones, BYOL-A creates contrasts in an augmented audio segment pair derived from a single audio segment. With a combination of normalization and augmentation techniques, BYOL-A achieves state-of-the-art results in various downstream tasks. Extensive ablation studies also clarified the contribution of each component and their combinations.

Related papers

• AudioFormer: Audio Transformer learns audio feature representations from discrete acoustic codes [6.375996974877916]
  We propose a method named AudioFormer, which learns audio feature representations through the acquisition of discrete acoustic codes. Our research outcomes demonstrate that AudioFormer attains significantly improved performance compared to prevailing monomodal audio classification models.
  arXiv  Detail & Related papers  (2023-08-14T15:47:25Z)
• Language-Guided Audio-Visual Source Separation via Trimodal Consistency [64.0580750128049]
  A key challenge in this task is learning to associate the linguistic description of a sound-emitting object to its visual features and the corresponding components of the audio waveform. We adapt off-the-shelf vision-language foundation models to provide pseudo-target supervision via two novel loss functions. We demonstrate the effectiveness of our self-supervised approach on three audio-visual separation datasets.
  arXiv  Detail & Related papers  (2023-03-28T22:45:40Z)
• MAViL: Masked Audio-Video Learners [68.61844803682145]
  We present Masked Audio-Video learners (MAViL) to train audio-visual representations. Pre-training with MAViL enables the model to perform well in audio-visual classification and retrieval tasks. For the first time, a self-supervised audio-visual model outperforms ones that use external supervision on benchmarks.
  arXiv  Detail & Related papers  (2022-12-15T18:59:59Z)
• Self-supervised Contrastive Learning for Audio-Visual Action Recognition [7.188231323934023]
  The underlying correlation between audio and visual modalities can be utilized to learn supervised information for unlabeled videos. We propose an end-to-end self-supervised framework named Audio-Visual Contrastive Learning (A), to learn discriminative audio-visual representations for action recognition.
  arXiv  Detail & Related papers  (2022-04-28T10:01:36Z)
• Conformer-Based Self-Supervised Learning for Non-Speech Audio Tasks [20.316239155843963]
  We propose a self-supervised audio representation learning method and apply it to a variety of downstream non-speech audio tasks. On the AudioSet benchmark, we achieve a mean average precision (mAP) score of 0.415, which is a new state-of-the-art on this dataset.
  arXiv  Detail & Related papers  (2021-10-14T12:32:40Z)
• Contrastive Learning of General-Purpose Audio Representations [33.15189569532155]
  We introduce COLA, a self-supervised pre-training approach for learning a general-purpose representation of audio. We build on recent advances in contrastive learning for computer vision and reinforcement learning to design a lightweight, easy-to-implement model of audio.
  arXiv  Detail & Related papers  (2020-10-21T11:56:22Z)
• Self-Supervised Learning of Audio-Visual Objects from Video [108.77341357556668]
  We introduce a model that uses attention to localize and group sound sources, and optical flow to aggregate information over time. We demonstrate the effectiveness of the audio-visual object embeddings that our model learns by using them for four downstream speech-oriented tasks.
  arXiv  Detail & Related papers  (2020-08-10T16:18:01Z)
• Learning Speech Representations from Raw Audio by Joint Audiovisual Self-Supervision [63.564385139097624]
  We propose a method to learn self-supervised speech representations from the raw audio waveform. We train a raw audio encoder by combining audio-only self-supervision (by predicting informative audio attributes) with visual self-supervision (by generating talking faces from audio) Our results demonstrate the potential of multimodal self-supervision in audiovisual speech for learning good audio representations.
  arXiv  Detail & Related papers  (2020-07-08T14:07:06Z)
• Deep Audio-Visual Learning: A Survey [53.487938108404244]
  We divide the current audio-visual learning tasks into four different subfields. We discuss state-of-the-art methods as well as the remaining challenges of each subfield. We summarize the commonly used datasets and performance metrics.
  arXiv  Detail & Related papers  (2020-01-14T13:11:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.