Mask-Weighted Spatial Likelihood Coding for Speaker-Independent Joint Localization and Mask Estimation

• URL: http://arxiv.org/abs/2410.19595v1
• Date: Fri, 25 Oct 2024 14:43:32 GMT
• Title: Mask-Weighted Spatial Likelihood Coding for Speaker-Independent Joint Localization and Mask Estimation
• Authors: Jakob Kienegger, Alina Mannanova, Timo Gerkmann,
• Abstract summary: Time-frequency masks and relative directions of the speakers regarding a fixed spatial grid can be used to estimate the beamformer's parameters. We analyze how to encode both mask and positioning into such a grid to enable joint estimation of both quantities.
• Score: 14.001679439460359
• License:
• Abstract: Due to their robustness and flexibility, neural-driven beamformers are a popular choice for speech separation in challenging environments with a varying amount of simultaneous speakers alongside noise and reverberation. Time-frequency masks and relative directions of the speakers regarding a fixed spatial grid can be used to estimate the beamformer's parameters. To some degree, speaker-independence is achieved by ensuring a greater amount of spatial partitions than speech sources. In this work, we analyze how to encode both mask and positioning into such a grid to enable joint estimation of both quantities. We propose mask-weighted spatial likelihood coding and show that it achieves considerable performance in both tasks compared to baseline encodings optimized for either localization or mask estimation. In the same setup, we demonstrate superiority for joint estimation of both quantities. Conclusively, we propose a universal approach which can replace an upstream sound source localization system solely by adapting the training framework, making it highly relevant in performance-critical scenarios.

Related papers

• A unified multichannel far-field speech recognition system: combining neural beamforming with attention based end-to-end model [14.795953417531907]
  We propose a unified multichannel far-field speech recognition system that combines the neural beamforming and transformer-based Listen, Spell, Attend (LAS) speech recognition system. The proposed method achieve 19.26% improvement when compared with a strong baseline.
  arXiv  Detail & Related papers  (2024-01-05T07:11:13Z)
• DASA: Difficulty-Aware Semantic Augmentation for Speaker Verification [55.306583814017046]
  We present a novel difficulty-aware semantic augmentation (DASA) approach for speaker verification. DASA generates diversified training samples in speaker embedding space with negligible extra computing cost. The best result achieves a 14.6% relative reduction in EER metric on CN-Celeb evaluation set.
  arXiv  Detail & Related papers  (2023-10-18T17:07:05Z)
• Variable Attention Masking for Configurable Transformer Transducer Speech Recognition [23.546294634238677]
  We study the use of attention masking in transformer transducer based speech recognition. We show that chunked masking achieves a better accuracy vs latency trade-off compared to fixed masking. We also show that variable masking improves the accuracy by up to 8% relative in the acoustic re-scoring scenario.
  arXiv  Detail & Related papers  (2022-11-02T19:14:02Z)
• Multi-scale Speaker Diarization with Dynamic Scale Weighting [14.473173007997751]
  We propose a more advanced multi-scale diarization system based on a multi-scale diarization decoder. Our proposed system achieves a state-of-art performance on the CALLHOME and AMI MixHeadset datasets, with 3.92% and 1.05% diarization error rates, respectively.
  arXiv  Detail & Related papers  (2022-03-30T01:26:31Z)
• Speaker Embedding-aware Neural Diarization: a Novel Framework for Overlapped Speech Diarization in the Meeting Scenario [51.5031673695118]
  We reformulate overlapped speech diarization as a single-label prediction problem. We propose the speaker embedding-aware neural diarization (SEND) system.
  arXiv  Detail & Related papers  (2022-03-18T06:40:39Z)
• Diarisation using location tracking with agglomerative clustering [42.13772744221499]
  This paper explicitly models the movements of speakers within an Agglomerative Hierarchical Clustering (AHC) diarisation framework. Experiments show that the proposed approach is able to yield improvements on a Microsoft rich meeting transcription task.
  arXiv  Detail & Related papers  (2021-09-22T08:54:10Z)
• Data Fusion for Audiovisual Speaker Localization: Extending Dynamic Stream Weights to the Spatial Domain [103.3388198420822]
  Esting the positions of multiple speakers can be helpful for tasks like automatic speech recognition or speaker diarization. This paper proposes a novel audiovisual data fusion framework for speaker localization by assigning individual dynamic stream weights to specific regions. A performance evaluation using audiovisual recordings yields promising results, with the proposed fusion approach outperforming all baseline models.
  arXiv  Detail & Related papers  (2021-02-23T09:59:31Z)
• End-to-End Speaker Diarization as Post-Processing [64.12519350944572]
  Clustering-based diarization methods partition frames into clusters of the number of speakers. Some end-to-end diarization methods can handle overlapping speech by treating the problem as multi-label classification. We propose to use a two-speaker end-to-end diarization method as post-processing of the results obtained by a clustering-based method.
  arXiv  Detail & Related papers  (2020-12-18T05:31:07Z)
• Any-to-Many Voice Conversion with Location-Relative Sequence-to-Sequence Modeling [61.351967629600594]
  This paper proposes an any-to-many location-relative, sequence-to-sequence (seq2seq), non-parallel voice conversion approach. In this approach, we combine a bottle-neck feature extractor (BNE) with a seq2seq synthesis module. Objective and subjective evaluations show that the proposed any-to-many approach has superior voice conversion performance in terms of both naturalness and speaker similarity.
  arXiv  Detail & Related papers  (2020-09-06T13:01:06Z)
• Temporal-Spatial Neural Filter: Direction Informed End-to-End Multi-channel Target Speech Separation [66.46123655365113]
  Target speech separation refers to extracting the target speaker's speech from mixed signals. Two main challenges are the complex acoustic environment and the real-time processing requirement. We propose a temporal-spatial neural filter, which directly estimates the target speech waveform from multi-speaker mixture.
  arXiv  Detail & Related papers  (2020-01-02T11:12:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.