Predictive-Generative Drift Decomposition for Speech Enhancement and Separation

• URL: http://arxiv.org/abs/2605.06189v1
• Date: Thu, 07 May 2026 13:04:38 GMT
• Title: Predictive-Generative Drift Decomposition for Speech Enhancement and Separation
• Abstract summary: We propose a plug-and-play framework for speech enhancement and separation that augments predictive methods with a generative speech prior.<n>We decompose expressive dynamics into a task-specific drift and a denoising component, allowing a predictive estimate to be integrated directly into the generative sampling process.<n>This results in a mathematically grounded framework for combining strong pretrained predictors with the power of generative models.
• Score: 42.49632892186173
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We propose a plug-and-play framework for speech enhancement and separation that augments predictive methods with a generative speech prior. Our approach, termed Stochastic Interpolant Prior for Speech (SIPS), builds on stochastic interpolants and leverages their flexibility to bridge predictive and generative modeling. Specifically, we decompose the interpolation dynamics into a task-specific drift and a stochastic denoising component, allowing a predictive estimate to be integrated directly into the generative sampling process. This results in a mathematically grounded framework for combining strong pretrained predictors with the expressive power of generative models. To this end, we train a score model using only clean speech, yielding a degradation-agnostic prior that can be reused across tasks. During inference, the predictor provides a deterministic drift that steers the sampling process toward a task-consistent estimate, while the score model preserves perceptual naturalness. Unlike prior hybrid approaches, which typically rely on architecture-specific conditioning and are tied to particular predictors or degradation settings, SIPS provides a unified framework that generalizes across predictors and additive degradation tasks. We demonstrate its effectiveness for both speech enhancement and speech separation using recent predictors such as SEMamba and FlexIO. The proposed method consistently improves perceptual quality, achieving gains up +1.0 NISQA for speech separation.

Related papers

• AdaMTP: An Adaptive Training Paradigm for Multi-Token Prediction [32.62594861280837]
  Multi-Token Prediction (MTP) has emerged as an effective paradigm that augments a shared Large Language Model backbone with auxiliary heads.<n>We propose AdaMTP, an adaptive training paradigm that aligns the prediction horizon with the intrinsic predictability of the sequence.<n>Across mathematical reasoning, code generation, and general benchmarks on three backbones, AdaMTP consistently outperforms standard MTP in both task performance and inference speedup.
  arXiv  Detail & Related papers  (2026-08-01T04:18:49Z)
• Joint Model and Data Sparsification via the Marginal Likelihood [53.29070892356214]
  We propose jointly learning individual feature and sample relevancies, enabling simultaneous model and data sparsification.<n>This symmetric pruning of model and data offers a natural extension that preserves conjugacy.<n> Empirical results across diverse regression tasks affirm that a joint ARD approach consistently yields both sparse and robust prediction models.
  arXiv  Detail & Related papers  (2026-05-28T13:26:53Z)
• Factorize to Generalize: Retrieval-Guided Invariant-Dynamic Decomposition for Time Series Forecasting [64.2640656619583]
  Time series foundation models (TSFMs) have recently achieved strong zero-shot forecasting performance through large-scale pretraining and retrieval-augmented prediction.<n>We propose a Retrieval-guided Invariant-Dynamic DEcomposition framework for time series forecasting.
  arXiv  Detail & Related papers  (2026-05-24T07:31:55Z)
• Perturbation is All You Need for Extrapolating Language Models [10.671612115533597]
  We introduce a simple yet powerful framework for training large language models.<n>We develop a rigorous theory of extrapolability, namely, the capacity of a model class to make reliable predictions.<n>Results show that the proposed method consistently improves out-of-support prediction while maintaining competitive in-support performance.
  arXiv  Detail & Related papers  (2026-05-05T23:03:33Z)
• Anchored Variational Inference for Personalized Sequential Latent-State Models [0.0]
  Sequential latent-variable models with subject-specific random effects provide a flexible framework for modeling temporally structured data.<n>We propose an anchored variational inference framework for efficient approximate inference in this setting.<n>We show that the posterior mean is a nearly optimal anchor point and that the resulting anchored variational EM algorithm approximately preserves the local monotonicity behavior of standard variational inference.
  arXiv  Detail & Related papers  (2026-04-25T22:02:48Z)
• SynCast: Synergizing Contradictions in Precipitation Nowcasting via Diffusion Sequential Preference Optimization [62.958457694151384]
  We introduce preference optimization into precipitation nowcasting for the first time, motivated by the success of reinforcement learning from human feedback in large language models.<n>In the first stage, the framework focuses on reducing FAR, training the model to effectively suppress false alarms.
  arXiv  Detail & Related papers  (2025-10-22T16:11:22Z)
• Self-Boost via Optimal Retraining: An Analysis via Approximate Message Passing [58.52119063742121]
  Retraining a model using its own predictions together with the original, potentially noisy labels is a well-known strategy for improving the model performance.<n>This paper addresses the question of how to optimally combine the model's predictions and the provided labels.<n>Our main contribution is the derivation of the Bayes optimal aggregator function to combine the current model's predictions and the given labels.
  arXiv  Detail & Related papers  (2025-05-21T07:16:44Z)
• Causal Self-supervised Pretrained Frontend with Predictive Code for Speech Separation [42.63061599979695]
  Speech separation (SS) seeks to disentangle a multi-talker speech mixture into single-talker speech streams.<n> Causal separation models, which rely only on past and present information, offer a promising solution for real-time streaming.<n>We introduce a novel that is designed to mitigate the mismatch between training and run-time inference by implicitly incorporating future information into causal models.
  arXiv  Detail & Related papers  (2025-04-03T06:18:30Z)
• Skeleton2vec: A Self-supervised Learning Framework with Contextualized Target Representations for Skeleton Sequence [56.092059713922744]
  We show that using high-level contextualized features as prediction targets can achieve superior performance. Specifically, we propose Skeleton2vec, a simple and efficient self-supervised 3D action representation learning framework. Our proposed Skeleton2vec outperforms previous methods and achieves state-of-the-art results.
  arXiv  Detail & Related papers  (2024-01-01T12:08:35Z)
• Fine-grained Forecasting Models Via Gaussian Process Blurring Effect [6.472434306724611]
  Time series forecasting is a challenging task due to the existence of complex and dynamic temporal dependencies. Using more training data is one way to improve the accuracy, but this source is often limited. We are building on successful denoising approaches for image generation by advocating for an end-to-end forecasting and denoising paradigm.
  arXiv  Detail & Related papers  (2023-12-21T20:25:16Z)
• Fast and efficient speech enhancement with variational autoencoders [0.0]
  Unsupervised speech enhancement based on variational autoencoders has shown promising performance compared with the commonly used supervised methods. We propose a new approach based on Langevin dynamics that generates multiple sequences of samples and comes with a total variation-based regularization to incorporate temporal correlations of latent vectors. Our experiments demonstrate that the developed framework makes an effective compromise between computational efficiency and enhancement quality, and outperforms existing methods.
  arXiv  Detail & Related papers  (2022-11-02T09:52:13Z)
• Discretization and Re-synthesis: an alternative method to solve the Cocktail Party Problem [65.25725367771075]
  This study demonstrates, for the first time, that the synthesis-based approach can also perform well on this problem. Specifically, we propose a novel speech separation/enhancement model based on the recognition of discrete symbols. By utilizing the synthesis model with the input of discrete symbols, after the prediction of discrete symbol sequence, each target speech could be re-synthesized.
  arXiv  Detail & Related papers  (2021-12-17T08:35:40Z)
• Generating diverse and natural text-to-speech samples using a quantized fine-grained VAE and auto-regressive prosody prior [53.69310441063162]
  This paper proposes a sequential prior in a discrete latent space which can generate more naturally sounding samples. We evaluate the approach using listening tests, objective metrics of automatic speech recognition (ASR) performance, and measurements of prosody attributes.
  arXiv  Detail & Related papers  (2020-02-06T12:35:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.