Leveraging Spatial Cues from Cochlear Implant Microphones to Efficiently Enhance Speech Separation in Real-World Listening Scenes

• URL: http://arxiv.org/abs/2501.14610v1
• Date: Fri, 24 Jan 2025 16:30:58 GMT
• Title: Leveraging Spatial Cues from Cochlear Implant Microphones to Efficiently Enhance Speech Separation in Real-World Listening Scenes
• Authors: Feyisayo Olalere, Kiki van der Heijden, Christiaan H. Stronks, Jeroen Briaire, Johan HM Frijns, Marcel van Gerven,
• Abstract summary: We quantify the impact of real-world acoustic scenes on speech separation and explore how spatial cues can enhance separation quality efficiently. Our findings show that spatial cues (both implicit and explicit) improve separation for mixtures with spatially separated talkers. Explicit spatial cues are particularly beneficial when implicit spatial cues are weak. These results emphasize the importance of training models on real-world data to improve generalizability in everyday listening scenarios.
• Score: 1.1081718316044291
• License:
• Abstract: Speech separation approaches for single-channel, dry speech mixtures have significantly improved. However, real-world spatial and reverberant acoustic environments remain challenging, limiting the effectiveness of these approaches for assistive hearing devices like cochlear implants (CIs). To address this, we quantify the impact of real-world acoustic scenes on speech separation and explore how spatial cues can enhance separation quality efficiently. We analyze performance based on implicit spatial cues (inherent in the acoustic input and learned by the model) and explicit spatial cues (manually calculated spatial features added as auxiliary inputs). Our findings show that spatial cues (both implicit and explicit) improve separation for mixtures with spatially separated and nearby talkers. Furthermore, spatial cues enhance separation when spectral cues are ambiguous, such as when voices are similar. Explicit spatial cues are particularly beneficial when implicit spatial cues are weak. For instance, single CI microphone recordings provide weaker implicit spatial cues than bilateral CIs, but even single CIs benefit from explicit cues. These results emphasize the importance of training models on real-world data to improve generalizability in everyday listening scenarios. Additionally, our statistical analyses offer insights into how data properties influence model performance, supporting the development of efficient speech separation approaches for CIs and other assistive devices in real-world settings.

Related papers

• Attention-Driven Multichannel Speech Enhancement in Moving Sound Source Scenarios [11.811571392419324]
  Speech enhancement algorithms typically assume a stationary sound source, a common mismatch with reality that limits their performance in real-world scenarios. This paper focuses on attention-driven spatial filtering techniques designed for dynamic settings.
  arXiv  Detail & Related papers  (2023-12-17T16:12:35Z)
• Exploring Self-Supervised Contrastive Learning of Spatial Sound Event Representation [21.896817015593122]
  MC-SimCLR learns joint spectral and spatial representations from unlabeled spatial audios. We propose a multi-level data augmentation pipeline that augments different levels of audio features. We find that linear layers on top of the learned representation significantly outperform supervised models in terms of both event classification accuracy and localization error.
  arXiv  Detail & Related papers  (2023-09-27T18:23:03Z)
• Spectral Enhanced Rectangle Transformer for Hyperspectral Image Denoising [64.11157141177208]
  We propose a spectral enhanced rectangle Transformer to model the spatial and spectral correlation in hyperspectral images. For the former, we exploit the rectangle self-attention horizontally and vertically to capture the non-local similarity in the spatial domain. For the latter, we design a spectral enhancement module that is capable of extracting global underlying low-rank property of spatial-spectral cubes to suppress noise.
  arXiv  Detail & Related papers  (2023-04-03T09:42:13Z)
• Multi-Dimensional and Multi-Scale Modeling for Speech Separation Optimized by Discriminative Learning [9.84949849886926]
  Intra-SE-Conformer and Inter-Transformer (ISCIT) for speech separation. New network SE-Conformer can model audio sequences in multiple dimensions and scales.
  arXiv  Detail & Related papers  (2023-03-07T08:53:20Z)
• Spatial mixup: Directional loudness modification as data augmentation for sound event localization and detection [9.0259157539478]
  We propose Spatial Mixup as an application of parametric spatial audio effects for data augmentation. The modifications enhance or suppress signals arriving from certain directions, although the effect is less pronounced. The method is evaluated with experiments in the DCASE 2021 Task 3 dataset, where spatial mixup increases performance over a non-augmented baseline.
  arXiv  Detail & Related papers  (2021-10-12T16:16:58Z)
• Stepwise-Refining Speech Separation Network via Fine-Grained Encoding in High-order Latent Domain [34.23260020137834]
  We propose the Stepwise-Refining Speech Separation Network (SRSSN), which follows a coarse-to-fine separation framework. It first learns a 1-order latent domain to define an encoding space and thereby performs a rough separation in the coarse phase. It then learns a new latent domain along each basis function of the existing latent domain to obtain a high-order latent domain in the refining phase.
  arXiv  Detail & Related papers  (2021-10-10T13:21:16Z)
• Bridging the Gap Between Clean Data Training and Real-World Inference for Spoken Language Understanding [76.89426311082927]
  Existing models are trained on clean data, which causes a textitgap between clean data training and real-world inference. We propose a method from the perspective of domain adaptation, by which both high- and low-quality samples are embedding into similar vector space. Experiments on the widely-used dataset, Snips, and large scale in-house dataset (10 million training examples) demonstrate that this method not only outperforms the baseline models on real-world (noisy) corpus but also enhances the robustness, that is, it produces high-quality results under a noisy environment.
  arXiv  Detail & Related papers  (2021-04-13T17:54:33Z)
• Data Fusion for Audiovisual Speaker Localization: Extending Dynamic Stream Weights to the Spatial Domain [103.3388198420822]
  Esting the positions of multiple speakers can be helpful for tasks like automatic speech recognition or speaker diarization. This paper proposes a novel audiovisual data fusion framework for speaker localization by assigning individual dynamic stream weights to specific regions. A performance evaluation using audiovisual recordings yields promising results, with the proposed fusion approach outperforming all baseline models.
  arXiv  Detail & Related papers  (2021-02-23T09:59:31Z)
• Cross-domain Adaptation with Discrepancy Minimization for Text-independent Forensic Speaker Verification [61.54074498090374]
  This study introduces a CRSS-Forensics audio dataset collected in multiple acoustic environments. We pre-train a CNN-based network using the VoxCeleb data, followed by an approach which fine-tunes part of the high-level network layers with clean speech from CRSS-Forensics.
  arXiv  Detail & Related papers  (2020-09-05T02:54:33Z)
• Spatial and spectral deep attention fusion for multi-channel speech separation using deep embedding features [60.20150317299749]
  Multi-channel deep clustering (MDC) has acquired a good performance for speech separation. We propose a deep attention fusion method to dynamically control the weights of the spectral and spatial features and combine them deeply. Experimental results show that the proposed method outperforms MDC baseline and even better than the ideal binary mask (IBM)
  arXiv  Detail & Related papers  (2020-02-05T03:49:39Z)
• Temporal-Spatial Neural Filter: Direction Informed End-to-End Multi-channel Target Speech Separation [66.46123655365113]
  Target speech separation refers to extracting the target speaker's speech from mixed signals. Two main challenges are the complex acoustic environment and the real-time processing requirement. We propose a temporal-spatial neural filter, which directly estimates the target speech waveform from multi-speaker mixture.
  arXiv  Detail & Related papers  (2020-01-02T11:12:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.