Wavesplit: End-to-End Speech Separation by Speaker Clustering

• URL: http://arxiv.org/abs/2002.08933v2
• Date: Thu, 2 Jul 2020 13:57:33 GMT
• Title: Wavesplit: End-to-End Speech Separation by Speaker Clustering
• Abstract summary: We introduce Wavesplit, an end-to-end source separation system. From a single mixture, the model infers a representation for each source and then estimates each source signal. Wavesplit is also applicable to other domains, by separating fetal and maternal heart rates from a single abdominal electrocardiogram.
• Score: 35.573006591664864
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We introduce Wavesplit, an end-to-end source separation system. From a single mixture, the model infers a representation for each source and then estimates each source signal given the inferred representations. The model is trained to jointly perform both tasks from the raw waveform. Wavesplit infers a set of source representations via clustering, which addresses the fundamental permutation problem of separation. For speech separation, our sequence-wide speaker representations provide a more robust separation of long, challenging recordings compared to prior work. Wavesplit redefines the state-of-the-art on clean mixtures of 2 or 3 speakers (WSJ0-2/3mix), as well as in noisy and reverberated settings (WHAM/WHAMR). We also set a new benchmark on the recent LibriMix dataset. Finally, we show that Wavesplit is also applicable to other domains, by separating fetal and maternal heart rates from a single abdominal electrocardiogram.

Related papers

• SSNAPS: Audio-Visual Separation of Speech and Background Noise with Diffusion Inverse Sampling [23.130313134690443]
  This paper addresses the challenge of audio-visual single-microphone speech separation and enhancement in the presence of real-world environmental noise.<n>Our approach is based on generative inverse sampling, where we model clean speech and ambient noise with dedicated diffusion priors and jointly leverage them to recover all underlying sources.<n>We evaluate on mixtures of 1, 2, and 3 speakers with noise and show that, despite being entirely unsupervised, our method consistently outperforms leading supervised baselines in acWER across all conditions.
  arXiv  Detail & Related papers  (2026-02-01T18:57:53Z)
• High-Quality Sound Separation Across Diverse Categories via Visually-Guided Generative Modeling [65.02357548201188]
  We propose DAVIS, a Diffusion-based Audio-VIsual Separation framework that solves the audio-visual sound source separation task through generative learning.<n>Our framework operates by synthesizing the desired separated sound spectrograms directly from a noise distribution, conditioned concurrently on the mixed audio input and associated visual information.
  arXiv  Detail & Related papers  (2025-09-26T08:46:00Z)
• Separate This, and All of these Things Around It: Music Source Separation via Hyperellipsoidal Queries [53.30852012059025]
  Music source separation is an audio-to-audio retrieval task. Recent work in music source separation has begun to challenge the fixed-stem paradigm. We propose the use of hyperellipsoidal regions as queries to allow for an intuitive yet easily parametrizable approach to specifying both the target (location) and its spread.
  arXiv  Detail & Related papers  (2025-01-27T16:13:50Z)
• A Generalized Bandsplit Neural Network for Cinematic Audio Source Separation [39.45425155123186]
  We develop a model generalizing the Bandsplit RNN for any complete or overcomplete partitions of the frequency axis. A loss function motivated by the signal-to-noise ratio and the sparsity-promoting property of the 1-norm was proposed. Our best model sets the state of the art on the Divide and Remaster dataset with performance above the ideal ratio mask for the dialogue stem.
  arXiv  Detail & Related papers  (2023-09-05T19:19:22Z)
• UNSSOR: Unsupervised Neural Speech Separation by Leveraging Over-determined Training Mixtures [60.879679764741624]
  In reverberant conditions, each microphone acquires a mixture signal of multiple speakers at a different location. We propose UNSSOR, an algorithm for $textbfu$nsupervised $textbfn$eural. We show that this loss can promote unsupervised separation of speakers.
  arXiv  Detail & Related papers  (2023-05-31T17:28:02Z)
• Separate And Diffuse: Using a Pretrained Diffusion Model for Improving Source Separation [99.19786288094596]
  We show how the upper bound can be generalized to the case of random generative models. We show state-of-the-art results on 2, 3, 5, 10, and 20 speakers on multiple benchmarks.
  arXiv  Detail & Related papers  (2023-01-25T18:21:51Z)
• End-to-End Multi-speaker ASR with Independent Vector Analysis [80.83577165608607]
  We develop an end-to-end system for multi-channel, multi-speaker automatic speech recognition. We propose a paradigm for joint source separation and dereverberation based on the independent vector analysis (IVA) paradigm.
  arXiv  Detail & Related papers  (2022-04-01T05:45:33Z)
• Visual Scene Graphs for Audio Source Separation [65.47212419514761]
  State-of-the-art approaches for visually-guided audio source separation typically assume sources that have characteristic sounds, such as musical instruments. We propose Audio Visual Scene Graph Segmenter (AVSGS), a novel deep learning model that embeds the visual structure of the scene as a graph and segments this graph into subgraphs. Our pipeline is trained end-to-end via a self-supervised task consisting of separating audio sources using the visual graph from artificially mixed sounds.
  arXiv  Detail & Related papers  (2021-09-24T13:40:51Z)
• Squeeze-Excitation Convolutional Recurrent Neural Networks for Audio-Visual Scene Classification [4.191965713559235]
  This paper presents a multi-modal model for automatic scene classification. It exploits simultaneously auditory and visual information. It has been shown to provide an excellent trade-off between prediction performance and system complexity.
  arXiv  Detail & Related papers  (2021-07-28T06:10:10Z)
• Self-Supervised Learning from Automatically Separated Sound Scenes [38.71803524843168]
  This paper explores the use of unsupervised automatic sound separation to decompose unlabeled sound scenes into semantically-linked views. We find that learning to associate input mixtures with their automatically separated outputs yields stronger representations than past approaches.
  arXiv  Detail & Related papers  (2021-05-05T15:37:17Z)
• End-to-End Diarization for Variable Number of Speakers with Local-Global Networks and Discriminative Speaker Embeddings [66.50782702086575]
  We present an end-to-end deep network model that performs meeting diarization from single-channel audio recordings. The proposed system is designed to handle meetings with unknown numbers of speakers, using variable-number permutation-invariant cross-entropy based loss functions.
  arXiv  Detail & Related papers  (2021-05-05T14:55:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.