Time-Domain Speech Extraction with Spatial Information and Multi Speaker Conditioning Mechanism

• URL: http://arxiv.org/abs/2102.03762v1
• Date: Sun, 7 Feb 2021 10:11:49 GMT
• Title: Time-Domain Speech Extraction with Spatial Information and Multi Speaker Conditioning Mechanism
• Authors: Jisi Zhang, Catalin Zorila, Rama Doddipatla, Jon Barker
• Abstract summary: We present a novel multi-channel speech extraction system to simultaneously extract multiple clean individual sources from a mixture. The proposed method is built on an improved multi-channel time-domain speech separation network. Experiments on 2-channel WHAMR! data show that the proposed system improves by 9% relative the source separation performance over a strong multi-channel baseline.
• Score: 27.19635746008699
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this paper, we present a novel multi-channel speech extraction system to simultaneously extract multiple clean individual sources from a mixture in noisy and reverberant environments. The proposed method is built on an improved multi-channel time-domain speech separation network which employs speaker embeddings to identify and extract multiple targets without label permutation ambiguity. To efficiently inform the speaker information to the extraction model, we propose a new speaker conditioning mechanism by designing an additional speaker branch for receiving external speaker embeddings. Experiments on 2-channel WHAMR! data show that the proposed system improves by 9% relative the source separation performance over a strong multi-channel baseline, and it increases the speech recognition accuracy by more than 16% relative over the same baseline.

Related papers

• Spectron: Target Speaker Extraction using Conditional Transformer with Adversarial Refinement [17.645026729525462]
  We propose a transformer-based end-to-end model to extract a target speaker's speech from a mixed audio signal. Our experiments show that the use of a dual path transformer in the separator backbone along with proposed training paradigm improves the CNN baseline by $3.12$ dB points.
  arXiv  Detail & Related papers  (2024-09-02T16:11:12Z)
• End-to-end Multichannel Speaker-Attributed ASR: Speaker Guided Decoder and Input Feature Analysis [0.0]
  We present an end-to-end multichannel speaker-attributed automatic speech recognition (MC-SA-ASR) system that combines a Conformer-based encoder with multi-frame crosschannel attention and a speaker-attributed Transformer-based decoder.
  arXiv  Detail & Related papers  (2023-10-16T06:40:18Z)
• Multi-Dimensional and Multi-Scale Modeling for Speech Separation Optimized by Discriminative Learning [9.84949849886926]
  Intra-SE-Conformer and Inter-Transformer (ISCIT) for speech separation. New network SE-Conformer can model audio sequences in multiple dimensions and scales.
  arXiv  Detail & Related papers  (2023-03-07T08:53:20Z)
• MIMO-DBnet: Multi-channel Input and Multiple Outputs DOA-aware Beamforming Network for Speech Separation [55.533789120204055]
  We propose an end-to-end beamforming network for direction guided speech separation given merely the mixture signal. Specifically, we design a multi-channel input and multiple outputs architecture to predict the direction-of-arrival based embeddings and beamforming weights for each source.
  arXiv  Detail & Related papers  (2022-12-07T01:52:40Z)
• Audio-visual multi-channel speech separation, dereverberation and recognition [70.34433820322323]
  This paper proposes an audio-visual multi-channel speech separation, dereverberation and recognition approach. The advantage of the additional visual modality over using audio only is demonstrated on two neural dereverberation approaches. Experiments conducted on the LRS2 dataset suggest that the proposed audio-visual multi-channel speech separation, dereverberation and recognition system outperforms the baseline.
  arXiv  Detail & Related papers  (2022-04-05T04:16:03Z)
• Serialized Multi-Layer Multi-Head Attention for Neural Speaker Embedding [93.16866430882204]
  In prior works, frame-level features from one layer are aggregated to form an utterance-level representation. Inspired by the Transformer network, our proposed method utilizes the hierarchical architecture of stacked self-attention mechanisms. With more layers stacked, the neural network can learn more discriminative speaker embeddings.
  arXiv  Detail & Related papers  (2021-07-14T05:38:48Z)
• Single channel voice separation for unknown number of speakers under reverberant and noisy settings [106.48335929548875]
  We present a unified network for voice separation of an unknown number of speakers. The proposed approach is composed of several separation heads optimized together with a speaker classification branch. We present a new noisy and reverberant dataset of up to five different speakers speaking simultaneously.
  arXiv  Detail & Related papers  (2020-11-04T14:59:14Z)
• Multi-talker ASR for an unknown number of sources: Joint training of source counting, separation and ASR [91.87500543591945]
  We develop an end-to-end multi-talker automatic speech recognition system for an unknown number of active speakers. Our experiments show very promising performance in counting accuracy, source separation and speech recognition. Our system generalizes well to a larger number of speakers than it ever saw during training.
  arXiv  Detail & Related papers  (2020-06-04T11:25:50Z)
• Audio-visual Multi-channel Recognition of Overlapped Speech [79.21950701506732]
  This paper presents an audio-visual multi-channel overlapped speech recognition system featuring tightly integrated separation front-end and recognition back-end. Experiments suggest that the proposed multi-channel AVSR system outperforms the baseline audio-only ASR system by up to 6.81% (26.83% relative) and 22.22% (56.87% relative) absolute word error rate (WER) reduction on overlapped speech constructed using either simulation or replaying of the lipreading sentence 2 dataset respectively.
  arXiv  Detail & Related papers  (2020-05-18T10:31:19Z)
• Supervised Speaker Embedding De-Mixing in Two-Speaker Environment [37.27421131374047]
  Instead of separating a two-speaker signal in signal space like speech source separation, a speaker embedding de-mixing approach is proposed. The proposed approach separates different speaker properties from a two-speaker signal in embedding space.
  arXiv  Detail & Related papers  (2020-01-14T20:13:43Z)
• Robust Speaker Recognition Using Speech Enhancement And Attention Model [37.33388614967888]
  Instead of individually processing speech enhancement and speaker recognition, the two modules are integrated into one framework by a joint optimisation using deep neural networks. To increase robustness against noise, a multi-stage attention mechanism is employed to highlight the speaker related features learned from context information in time and frequency domain. The obtained results show that the proposed approach using speech enhancement and multi-stage attention models outperforms two strong baselines not using them in most acoustic conditions in our experiments.
  arXiv  Detail & Related papers  (2020-01-14T20:03:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.