FlowTSE: Target Speaker Extraction with Flow Matching

• URL: http://arxiv.org/abs/2505.14465v1
• Date: Tue, 20 May 2025 15:01:30 GMT
• Title: FlowTSE: Target Speaker Extraction with Flow Matching
• Abstract summary: FlowTSE is a simple yet effective TSE approach based on conditional flow matching.<n>For tasks where phase reconstruction is crucial, we propose a novel vocoder conditioned on the complex STFT of the mixed signal.
• Score: 16.054014378418316
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Target speaker extraction (TSE) aims to isolate a specific speaker's speech from a mixture using speaker enrollment as a reference. While most existing approaches are discriminative, recent generative methods for TSE achieve strong results. However, generative methods for TSE remain underexplored, with most existing approaches relying on complex pipelines and pretrained components, leading to computational overhead. In this work, we present FlowTSE, a simple yet effective TSE approach based on conditional flow matching. Our model receives an enrollment audio sample and a mixed speech signal, both represented as mel-spectrograms, with the objective of extracting the target speaker's clean speech. Furthermore, for tasks where phase reconstruction is crucial, we propose a novel vocoder conditioned on the complex STFT of the mixed signal, enabling improved phase estimation. Experimental results on standard TSE benchmarks show that FlowTSE matches or outperforms strong baselines.

Related papers

• Mask2Flow-TSE: Two-Stage Target Speaker Extraction with Masking and Flow Matching [17.73633095059539]
  Target speaker extraction (TSE) extracts the target speaker's voice from overlapping speech mixtures given a reference utterance.<n>Discriminative methods apply time-frequency masking for fast inference but often over-suppress the target signal.<n>generative methods synthesize high-quality speech at the cost of numerous iterative steps.<n>We propose Mask2Flow-TSE, a two-stage framework combining the strengths of both paradigms.
  arXiv  Detail & Related papers  (2026-03-13T09:40:13Z)
• AlphaFlowTSE: One-Step Generative Target Speaker Extraction via Conditional AlphaFlow [39.38826937846528]
  We present a one-step conditional generative model trained with a Jacobian-vector product (JVP)-free AlphaFlow objective.<n>Experiments on Libri2Mix and REAL-T confirm that AlphaFlowTSE improves target-speaker similarity and real-mixture for downstream automatic speech recognition (ASR)
  arXiv  Detail & Related papers  (2026-03-11T12:14:25Z)
• GenTSE: Enhancing Target Speaker Extraction via a Coarse-to-Fine Generative Language Model [35.12859489567766]
  We present GenTSE, a two-stage decoder-only generative LM approach for TSE.<n> separating semantics and acoustics stabilizes decoding and yields more faithful, content-aligned target speech.<n>Experiments on Libri2Mix show that GenTSE surpasses previous LM-based systems in speech quality, intelligibility, and speaker consistency.
  arXiv  Detail & Related papers  (2025-12-24T06:13:02Z)
• High-Fidelity Speech Enhancement via Discrete Audio Tokens [35.61634772862795]
  DAC-SE1 is a language model-based SE framework leveraging discrete high-resolution audio representations.<n>Our experiments show that DAC-SE1 surpasses state-of-the-art autoregressive SE methods on both objective perceptual metrics and in a MUSHRA human evaluation.
  arXiv  Detail & Related papers  (2025-10-02T16:38:05Z)
• DiFlow-TTS: Discrete Flow Matching with Factorized Speech Tokens for Low-Latency Zero-Shot Text-To-Speech [8.537791317883576]
  Zero-shot Text-to-Speech (TTS) aims to synthesize high-quality speech that mimics the voice of an unseen speaker using only a short reference sample.<n>Recent approaches based on language models, diffusion, and flow matching have shown promising results in zero-shot TTS, but still suffer from slow inference and repetition artifacts.<n>We introduce DiFlow-TTS, which is the first model to explore purely Discrete Flow Matching for speech synthesis.
  arXiv  Detail & Related papers  (2025-09-11T17:16:52Z)
• SoloSpeech: Enhancing Intelligibility and Quality in Target Speech Extraction through a Cascaded Generative Pipeline [29.85417427778784]
  SoloSpeech is a cascaded generative pipeline that integrates compression, extraction, reconstruction, and correction processes.<n>It achieves the new state-of-the-art intelligibility and quality in target speech extraction and speech separation tasks.
  arXiv  Detail & Related papers  (2025-05-25T21:00:48Z)
• OZSpeech: One-step Zero-shot Speech Synthesis with Learned-Prior-Conditioned Flow Matching [3.05024318465243]
  OZSpeech is the first TTS method to explore optimal transport conditional flow matching with one-step sampling.<n>Our approach operates on disentangled, factorized components of speech in token format, enabling accurate modeling of each speech attribute.<n> Experimental results show that our method achieves promising performance over existing methods in content accuracy, naturalness, prosody generation, and speaker style preservation.
  arXiv  Detail & Related papers  (2025-05-19T07:31:55Z)
• DMOSpeech: Direct Metric Optimization via Distilled Diffusion Model in Zero-Shot Speech Synthesis [12.310318928818546]
  We introduce DMOSpeech, a distilled diffusion-based TTS model that achieves both faster inference and superior performance compared to its teacher model.<n>Our comprehensive experiments, validated through extensive human evaluation, show significant improvements in naturalness, intelligibility, and speaker similarity while reducing inference time by orders of magnitude.<n>This work establishes a new framework for aligning speech synthesis with human auditory preferences through direct metric optimization.
  arXiv  Detail & Related papers  (2024-10-14T21:17:58Z)
• Qifusion-Net: Layer-adapted Stream/Non-stream Model for End-to-End Multi-Accent Speech Recognition [1.0690007351232649]
  We propose a layer-adapted fusion (LAF) model, called Qifusion-Net, which does not require any prior knowledge about the target accent. Experiment results demonstrate that our proposed methods outperform the baseline with relative reductions of 22.1$%$ and 17.2$%$ in character error rate (CER) across multi accent test datasets.
  arXiv  Detail & Related papers  (2024-07-03T11:35:52Z)
• DDTSE: Discriminative Diffusion Model for Target Speech Extraction [62.422291953387955]
  We introduce the Discriminative Diffusion model for Target Speech Extraction (DDTSE) We apply the same forward process as diffusion models and utilize the reconstruction loss similar to discriminative methods. We devise a two-stage training strategy to emulate the inference process during model training.
  arXiv  Detail & Related papers  (2023-09-25T04:58:38Z)
• Diffusion-based speech enhancement with a weighted generative-supervised learning loss [0.0]
  Diffusion-based generative models have recently gained attention in speech enhancement (SE) We propose augmenting the original diffusion training objective with a mean squared error (MSE) loss, measuring the discrepancy between estimated enhanced speech and ground-truth clean speech.
  arXiv  Detail & Related papers  (2023-09-19T09:13:35Z)
• DiffSED: Sound Event Detection with Denoising Diffusion [70.18051526555512]
  We reformulate the SED problem by taking a generative learning perspective. Specifically, we aim to generate sound temporal boundaries from noisy proposals in a denoising diffusion process. During training, our model learns to reverse the noising process by converting noisy latent queries to the groundtruth versions.
  arXiv  Detail & Related papers  (2023-08-14T17:29:41Z)
• Isolation and Impartial Aggregation: A Paradigm of Incremental Learning without Interference [61.11137714507445]
  This paper focuses on the prevalent performance imbalance in the stages of incremental learning. We propose a stage-isolation based incremental learning framework. We evaluate the proposed method on four large benchmarks.
  arXiv  Detail & Related papers  (2022-11-29T06:57:48Z)
• Speaker Embedding-aware Neural Diarization: a Novel Framework for Overlapped Speech Diarization in the Meeting Scenario [51.5031673695118]
  We reformulate overlapped speech diarization as a single-label prediction problem. We propose the speaker embedding-aware neural diarization (SEND) system.
  arXiv  Detail & Related papers  (2022-03-18T06:40:39Z)
• Discretization and Re-synthesis: an alternative method to solve the Cocktail Party Problem [65.25725367771075]
  This study demonstrates, for the first time, that the synthesis-based approach can also perform well on this problem. Specifically, we propose a novel speech separation/enhancement model based on the recognition of discrete symbols. By utilizing the synthesis model with the input of discrete symbols, after the prediction of discrete symbol sequence, each target speech could be re-synthesized.
  arXiv  Detail & Related papers  (2021-12-17T08:35:40Z)
• A Study on Speech Enhancement Based on Diffusion Probabilistic Model [63.38586161802788]
  We propose a diffusion probabilistic model-based speech enhancement model (DiffuSE) model that aims to recover clean speech signals from noisy signals. The experimental results show that DiffuSE yields performance that is comparable to related audio generative models on the standardized Voice Bank corpus task.
  arXiv  Detail & Related papers  (2021-07-25T19:23:18Z)
• Simultaneous Denoising and Dereverberation Using Deep Embedding Features [64.58693911070228]
  We propose a joint training method for simultaneous speech denoising and dereverberation using deep embedding features. At the denoising stage, the DC network is leveraged to extract noise-free deep embedding features. At the dereverberation stage, instead of using the unsupervised K-means clustering algorithm, another neural network is utilized to estimate the anechoic speech.
  arXiv  Detail & Related papers  (2020-04-06T06:34:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.