TIGER: Time-frequency Interleaved Gain Extraction and Reconstruction for Efficient Speech Separation

• URL: http://arxiv.org/abs/2410.01469v1
• Date: Wed, 2 Oct 2024 12:21:06 GMT
• Title: TIGER: Time-frequency Interleaved Gain Extraction and Reconstruction for Efficient Speech Separation
• Authors: Mohan Xu, Kai Li, Guo Chen, Xiaolin Hu,
• Abstract summary: We propose a speech separation model with significantly reduced parameters and computational costs. TIGER leverages prior knowledge to divide frequency bands and compresses frequency information. We show that TIGER achieves performance surpassing state-of-the-art (SOTA) model TF-GridNet.
• Score: 19.126525226518975
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In recent years, much speech separation research has focused primarily on improving model performance. However, for low-latency speech processing systems, high efficiency is equally important. Therefore, we propose a speech separation model with significantly reduced parameters and computational costs: Time-frequency Interleaved Gain Extraction and Reconstruction network (TIGER). TIGER leverages prior knowledge to divide frequency bands and compresses frequency information. We employ a multi-scale selective attention module to extract contextual features, while introducing a full-frequency-frame attention module to capture both temporal and frequency contextual information. Additionally, to more realistically evaluate the performance of speech separation models in complex acoustic environments, we introduce a dataset called EchoSet. This dataset includes noise and more realistic reverberation (e.g., considering object occlusions and material properties), with speech from two speakers overlapping at random proportions. Experimental results showed that models trained on EchoSet had better generalization ability than those trained on other datasets to the data collected in the physical world, which validated the practical value of the EchoSet. On EchoSet and real-world data, TIGER significantly reduces the number of parameters by 94.3% and the MACs by 95.3% while achieving performance surpassing state-of-the-art (SOTA) model TF-GridNet. This is the first speech separation model with fewer than 1 million parameters that achieves performance comparable to the SOTA model.

Related papers

• Real Acoustic Fields: An Audio-Visual Room Acoustics Dataset and Benchmark [65.79402756995084]
  Real Acoustic Fields (RAF) is a new dataset that captures real acoustic room data from multiple modalities. RAF is the first dataset to provide densely captured room acoustic data.
  arXiv  Detail & Related papers  (2024-03-27T17:59:56Z)
• RTFS-Net: Recurrent Time-Frequency Modelling for Efficient Audio-Visual Speech Separation [18.93255531121519]
  We present a novel time-frequency domain audio-visual speech separation method. RTFS-Net applies its algorithms on the complex time-frequency bins yielded by the Short-Time Fourier Transform. This is the first time-frequency domain audio-visual speech separation method to outperform all contemporary time-domain counterparts.
  arXiv  Detail & Related papers  (2023-09-29T12:38:00Z)
• Audio-Visual Efficient Conformer for Robust Speech Recognition [91.3755431537592]
  We propose to improve the noise of the recently proposed Efficient Conformer Connectionist Temporal Classification architecture by processing both audio and visual modalities. Our experiments show that using audio and visual modalities allows to better recognize speech in the presence of environmental noise and significantly accelerate training, reaching lower WER with 4 times less training steps.
  arXiv  Detail & Related papers  (2023-01-04T05:36:56Z)
• Exploring Self-Attention Mechanisms for Speech Separation [11.210834842425955]
  This paper studies in-depth Transformers for speech separation. We extend our previous findings on the SepFormer by providing results on more challenging noisy and noisy-reverberant datasets. Finally, we investigate, for the first time in speech separation, the use of efficient self-attention mechanisms such as Linformers, Lonformers, and ReFormers.
  arXiv  Detail & Related papers  (2022-02-06T23:13:27Z)
• Improving Noise Robustness of Contrastive Speech Representation Learning with Speech Reconstruction [109.44933866397123]
  Noise robustness is essential for deploying automatic speech recognition systems in real-world environments. We employ a noise-robust representation learned by a refined self-supervised framework for noisy speech recognition. We achieve comparable performance to the best supervised approach reported with only 16% of labeled data.
  arXiv  Detail & Related papers  (2021-10-28T20:39:02Z)
• Raw Waveform Encoder with Multi-Scale Globally Attentive Locally Recurrent Networks for End-to-End Speech Recognition [45.858039215825656]
  We propose a new encoder that adopts globally attentive locally recurrent (GALR) networks and directly takes raw waveform as input. Experiments are conducted on a benchmark dataset AISHELL-2 and two large-scale Mandarin speech corpus of 5,000 hours and 21,000 hours.
  arXiv  Detail & Related papers  (2021-06-08T12:12:33Z)
• MixSpeech: Data Augmentation for Low-resource Automatic Speech Recognition [54.84624870942339]
  MixSpeech is a simple yet effective data augmentation method based on mixup for automatic speech recognition (ASR) We apply MixSpeech on two popular end-to-end speech recognition models including LAS (Listen, Attend and Spell) and Transformer. Experimental results show that MixSpeech achieves better accuracy than the baseline models without data augmentation.
  arXiv  Detail & Related papers  (2021-02-25T03:40:43Z)
• Rethinking Evaluation in ASR: Are Our Models Robust Enough? [30.114009549372923]
  We show that, in general, reverberative and additive noise augmentation improves generalization performance across domains. We demonstrate that when a large enough set of benchmarks is used, average word error rate (WER) performance over them provides a good proxy for performance on real-world noisy data.
  arXiv  Detail & Related papers  (2020-10-22T14:01:32Z)
• Real Time Speech Enhancement in the Waveform Domain [99.02180506016721]
  We present a causal speech enhancement model working on the raw waveform that runs in real-time on a laptop CPU. The proposed model is based on an encoder-decoder architecture with skip-connections. It is capable of removing various kinds of background noise including stationary and non-stationary noises.
  arXiv  Detail & Related papers  (2020-06-23T09:19:13Z)
• Temporal-Spatial Neural Filter: Direction Informed End-to-End Multi-channel Target Speech Separation [66.46123655365113]
  Target speech separation refers to extracting the target speaker's speech from mixed signals. Two main challenges are the complex acoustic environment and the real-time processing requirement. We propose a temporal-spatial neural filter, which directly estimates the target speech waveform from multi-speaker mixture.
  arXiv  Detail & Related papers  (2020-01-02T11:12:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.