DOA-Aware Audio-Visual Self-Supervised Learning for Sound Event Localization and Detection

• URL: http://arxiv.org/abs/2410.22803v1
• Date: Wed, 30 Oct 2024 08:31:58 GMT
• Title: DOA-Aware Audio-Visual Self-Supervised Learning for Sound Event Localization and Detection
• Authors: Yoto Fujita, Yoshiaki Bando, Keisuke Imoto, Masaki Onishi, Kazuyoshi Yoshii,
• Abstract summary: This paper describes sound event localization and detection (SELD) for spatial audio recordings captured by firstorder ambisonics (FOA) microphones. We propose a novel method of pretraining the feature extraction part of the deep neural network (DNN) in a self-supervised manner.
• Score: 16.92604848450722
• License:
• Abstract: This paper describes sound event localization and detection (SELD) for spatial audio recordings captured by firstorder ambisonics (FOA) microphones. In this task, one may train a deep neural network (DNN) using FOA data annotated with the classes and directions of arrival (DOAs) of sound events. However, the performance of this approach is severely bounded by the amount of annotated data. To overcome this limitation, we propose a novel method of pretraining the feature extraction part of the DNN in a self-supervised manner. We use spatial audio-visual recordings abundantly available as virtual reality contents. Assuming that sound objects are concurrently observed by the FOA microphones and the omni-directional camera, we jointly train audio and visual encoders with contrastive learning such that the audio and visual embeddings of the same recording and DOA are made close. A key feature of our method is that the DOA-wise audio embeddings are jointly extracted from the raw audio data, while the DOA-wise visual embeddings are separately extracted from the local visual crops centered on the corresponding DOA. This encourages the latent features of the audio encoder to represent both the classes and DOAs of sound events. The experiment using the DCASE2022 Task 3 dataset of 20 hours shows non-annotated audio-visual recordings of 100 hours reduced the error score of SELD from 36.4 pts to 34.9 pts.

Related papers

• Towards Open-Vocabulary Audio-Visual Event Localization [59.23161248808759]
  We introduce the Open-Vocabulary Audio-Visual Event localization problem. This problem requires localizing audio-visual events and predicting explicit categories for both seen and unseen data at inference. We propose the OV-AVEBench dataset, comprising 24,800 videos across 67 real-life audio-visual scenes.
  arXiv  Detail & Related papers  (2024-11-18T04:35:20Z)
• LAVSS: Location-Guided Audio-Visual Spatial Audio Separation [52.44052357829296]
  We propose a location-guided audio-visual spatial audio separator. The proposed LAVSS is inspired by the correlation between spatial audio and visual location. In addition, we adopt a pre-trained monaural separator to transfer knowledge from rich mono sounds to boost spatial audio separation.
  arXiv  Detail & Related papers  (2023-10-31T13:30:24Z)
• BAVS: Bootstrapping Audio-Visual Segmentation by Integrating Foundation Knowledge [43.92428145744478]
  We propose a two-stage bootstrapping audio-visual segmentation framework. In the first stage, we employ a segmentation model to localize potential sounding objects from visual data. In the second stage, we develop an audio-visual semantic integration strategy (AVIS) to localize the authentic-sounding objects.
  arXiv  Detail & Related papers  (2023-08-20T06:48:08Z)
• STARSS23: An Audio-Visual Dataset of Spatial Recordings of Real Scenes with Spatiotemporal Annotations of Sound Events [30.459545240265246]
  Sound events usually derive from visually source objects, e.g., sounds of come from the feet of a walker. This paper proposes an audio-visual sound event localization and detection (SELD) task. Audio-visual SELD systems can detect and localize sound events using signals from an array and audio-visual correspondence.
  arXiv  Detail & Related papers  (2023-06-15T13:37:14Z)
• Language-Guided Audio-Visual Source Separation via Trimodal Consistency [64.0580750128049]
  A key challenge in this task is learning to associate the linguistic description of a sound-emitting object to its visual features and the corresponding components of the audio waveform. We adapt off-the-shelf vision-language foundation models to provide pseudo-target supervision via two novel loss functions. We demonstrate the effectiveness of our self-supervised approach on three audio-visual separation datasets.
  arXiv  Detail & Related papers  (2023-03-28T22:45:40Z)
• Epic-Sounds: A Large-scale Dataset of Actions That Sound [64.24297230981168]
  Epic-Sounds is a large-scale dataset of audio annotations capturing temporal extents and class labels. We identify actions that can be discriminated purely from audio, through grouping these free-form descriptions of audio into classes. Overall, Epic-Sounds includes 78.4k categorised segments of audible events and actions, distributed across 44 classes as well as 39.2k non-categorised segments.
  arXiv  Detail & Related papers  (2023-02-01T18:19:37Z)
• Visually-Aware Audio Captioning With Adaptive Audio-Visual Attention [54.4258176885084]
  How to accurately recognize ambiguous sounds is a major challenge for audio captioning. We propose visually-aware audio captioning, which makes use of visual information to help the description of ambiguous sounding objects. Our proposed method achieves state-of-the-art results on machine translation metrics.
  arXiv  Detail & Related papers  (2022-10-28T22:45:41Z)
• Visual Scene Graphs for Audio Source Separation [65.47212419514761]
  State-of-the-art approaches for visually-guided audio source separation typically assume sources that have characteristic sounds, such as musical instruments. We propose Audio Visual Scene Graph Segmenter (AVSGS), a novel deep learning model that embeds the visual structure of the scene as a graph and segments this graph into subgraphs. Our pipeline is trained end-to-end via a self-supervised task consisting of separating audio sources using the visual graph from artificially mixed sounds.
  arXiv  Detail & Related papers  (2021-09-24T13:40:51Z)
• Exploiting Audio-Visual Consistency with Partial Supervision for Spatial Audio Generation [45.526051369551915]
  We propose an audio spatialization framework to convert a monaural video into a one exploiting the relationship across audio and visual components. Experiments on benchmark datasets confirm the effectiveness of our proposed framework in both semi-supervised and fully supervised scenarios.
  arXiv  Detail & Related papers  (2021-05-03T09:34:11Z)
• Self-supervised Neural Audio-Visual Sound Source Localization via Probabilistic Spatial Modeling [45.20508569656558]
  This paper presents a self-supervised training method using 360deg images and multichannel audio signals. By incorporating with the spatial information in multichannel audio signals, our method trains deep neural networks (DNNs) to distinguish multiple sound source objects. We also demonstrate that the visual DNN detected objects including talking visitors and specific exhibits from real data recorded in a science museum.
  arXiv  Detail & Related papers  (2020-07-28T03:52:53Z)
• Telling Left from Right: Learning Spatial Correspondence of Sight and Sound [16.99266133458188]
  We propose a novel self-supervised task to leverage a principle: matching spatial information in the audio stream to the positions of sound sources in the visual stream. We train a model to determine whether the left and right audio channels have been flipped, forcing it to reason about spatial localization across the visual and audio streams. We demonstrate that understanding spatial correspondence enables models to perform better on three audio-visual tasks, achieving quantitative gains over supervised and self-supervised baselines.
  arXiv  Detail & Related papers  (2020-06-11T04:00:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.