Learning Multi-Target TDOA Features for Sound Event Localization and Detection

• URL: http://arxiv.org/abs/2408.17166v1
• Date: Fri, 30 Aug 2024 10:09:12 GMT
• Title: Learning Multi-Target TDOA Features for Sound Event Localization and Detection
• Authors: Axel Berg, Johanna Engman, Jens Gulin, Karl Åström, Magnus Oskarsson,
• Abstract summary: We propose a new feature, based on neural generalized cross-correlations with phase-transform (NGCC-PHAT), that learns audio representations suitable for localization. We test our method on the STARSS23 dataset and demonstrate improved localization performance compared to using standard GCC-PHAT or SALSA-Lite input features.
• Score: 11.193111023459803
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Sound event localization and detection (SELD) systems using audio recordings from a microphone array rely on spatial cues for determining the location of sound events. As a consequence, the localization performance of such systems is to a large extent determined by the quality of the audio features that are used as inputs to the system. We propose a new feature, based on neural generalized cross-correlations with phase-transform (NGCC-PHAT), that learns audio representations suitable for localization. Using permutation invariant training for the time-difference of arrival (TDOA) estimation problem enables NGCC-PHAT to learn TDOA features for multiple overlapping sound events. These features can be used as a drop-in replacement for GCC-PHAT inputs to a SELD-network. We test our method on the STARSS23 dataset and demonstrate improved localization performance compared to using standard GCC-PHAT or SALSA-Lite input features.

Related papers

• MLCA-AVSR: Multi-Layer Cross Attention Fusion based Audio-Visual Speech Recognition [62.89464258519723]
  We propose a multi-layer cross-attention fusion based AVSR approach that promotes representation of each modality by fusing them at different levels of audio/visual encoders. Our proposed approach surpasses the first-place system, establishing a new SOTA cpCER of 29.13% on this dataset.
  arXiv  Detail & Related papers  (2024-01-07T08:59:32Z)
• Exploring Self-Supervised Contrastive Learning of Spatial Sound Event Representation [21.896817015593122]
  MC-SimCLR learns joint spectral and spatial representations from unlabeled spatial audios. We propose a multi-level data augmentation pipeline that augments different levels of audio features. We find that linear layers on top of the learned representation significantly outperform supervised models in terms of both event classification accuracy and localization error.
  arXiv  Detail & Related papers  (2023-09-27T18:23:03Z)
• Robust Feature Learning on Long-Duration Sounds for Acoustic Scene Classification [54.57150493905063]
  Acoustic scene classification (ASC) aims to identify the type of scene (environment) in which a given audio signal is recorded. We propose a robust feature learning (RFL) framework to train the CNN.
  arXiv  Detail & Related papers  (2021-08-11T03:33:05Z)
• DCASE 2021 Task 3: Spectrotemporally-aligned Features for Polyphonic Sound Event Localization and Detection [16.18806719313959]
  We propose a novel feature called spatial cue-augmented log-spectrogram (SALSA) with exact time-frequency mapping between the signal power and the source direction-of-arrival. We show that the deep learning-based models trained on this new feature outperformed the DCASE challenge baseline by a large margin.
  arXiv  Detail & Related papers  (2021-06-29T09:18:30Z)
• PILOT: Introducing Transformers for Probabilistic Sound Event Localization [107.78964411642401]
  This paper introduces a novel transformer-based sound event localization framework, where temporal dependencies in the received multi-channel audio signals are captured via self-attention mechanisms. The framework is evaluated on three publicly available multi-source sound event localization datasets and compared against state-of-the-art methods in terms of localization error and event detection accuracy.
  arXiv  Detail & Related papers  (2021-06-07T18:29:19Z)
• End-to-End Diarization for Variable Number of Speakers with Local-Global Networks and Discriminative Speaker Embeddings [66.50782702086575]
  We present an end-to-end deep network model that performs meeting diarization from single-channel audio recordings. The proposed system is designed to handle meetings with unknown numbers of speakers, using variable-number permutation-invariant cross-entropy based loss functions.
  arXiv  Detail & Related papers  (2021-05-05T14:55:29Z)
• Data Fusion for Audiovisual Speaker Localization: Extending Dynamic Stream Weights to the Spatial Domain [103.3388198420822]
  Esting the positions of multiple speakers can be helpful for tasks like automatic speech recognition or speaker diarization. This paper proposes a novel audiovisual data fusion framework for speaker localization by assigning individual dynamic stream weights to specific regions. A performance evaluation using audiovisual recordings yields promising results, with the proposed fusion approach outperforming all baseline models.
  arXiv  Detail & Related papers  (2021-02-23T09:59:31Z)
• Any-to-Many Voice Conversion with Location-Relative Sequence-to-Sequence Modeling [61.351967629600594]
  This paper proposes an any-to-many location-relative, sequence-to-sequence (seq2seq), non-parallel voice conversion approach. In this approach, we combine a bottle-neck feature extractor (BNE) with a seq2seq synthesis module. Objective and subjective evaluations show that the proposed any-to-many approach has superior voice conversion performance in terms of both naturalness and speaker similarity.
  arXiv  Detail & Related papers  (2020-09-06T13:01:06Z)
• C-SL: Contrastive Sound Localization with Inertial-Acoustic Sensors [5.101801159418222]
  We introduce contrastive sound localization (C-SL) with mobile inertial-acoustic sensor arrays of arbitrary geometry. C-SL learns mappings from acoustical measurements to an array-centered direction-of-arrival in a self-supervised manner. We believe the relaxed calibration process offered by C-SL paves the way toward truly personalized augmented hearing applications.
  arXiv  Detail & Related papers  (2020-06-09T06:36:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.