Repurposing a Speech Classifier for Guided Diffusion-Based Speech Generation

• URL: http://arxiv.org/abs/2606.20457v1
• Date: Thu, 18 Jun 2026 16:40:02 GMT
• Title: Repurposing a Speech Classifier for Guided Diffusion-Based Speech Generation
• Abstract summary: Speech classifier is repurposed as the backbone for diffusion generation.<n>A lightweight subnetwork is created that reuses intermediate representations and train only this subnetwork under a Denoising Score Matching objective.<n>Our work shows that a pretrained classifier can be repurposed for conditional generation, providing an appealing bridge between discriminative modeling and conditional speech.
• Score: 28.807557840756402
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Classifier guidance is a way to control diffusion generation by using a noise-conditioned classifier to steer the sampling process toward a target class. One drawback of classifier guidance is that it requires two separately trained models: a classifier and a diffusion model. We therefore study a more compact alternative in which a conventionally trained speech classifier is repurposed as the backbone for diffusion generation. Starting from a frozen noise-conditioned classifier in log-Mel space, we attach a lightweight subnetwork that reuses intermediate classifier representations and train only this subnetwork under a Denoising Score Matching objective. Our work shows that a pretrained classifier can be repurposed for conditional generation, providing an appealing bridge between discriminative modeling and conditional speech synthesis resulting in high speech quality within a single-backbone model, with reduced memory footprint and computational cost.

Related papers

• Joint Enhancement and Classification using Coupled Diffusion Models of Signals and Logits [45.99435851580745]
  We propose a general, domain-agnostic framework that integrates two interacting diffusion models.<n>We introduce three strategies to effectively model the joint distribution of the input and the logit.<n>The proposed framework surpasses traditional sequential enhancement baselines, delivering robust and flexible improvements in classification accuracy under diverse noise conditions.
  arXiv  Detail & Related papers  (2026-02-17T07:25:41Z)
• Generative Classifiers Avoid Shortcut Solutions [84.23247217037134]
  Discriminative approaches to classification often learn shortcuts that hold in-distribution but fail under minor distribution shift.<n>We show that generative classifiers can avoid this issue by modeling all features, both core and spurious, instead of mainly spurious ones.<n>We find that diffusion-based and autorerimigressive generative classifiers achieve state-of-the-art performance on five standard image and text distribution shift benchmarks.
  arXiv  Detail & Related papers  (2025-12-31T18:31:46Z)
• Studying Classifier(-Free) Guidance From a Classifier-Centric Perspective [100.54185280153753]
  We find that both classifier guidance and classifier-free guidance achieve conditional generation by pushing the denoising diffusion trajectories away from decision boundaries.<n>We propose a generic postprocessing step built upon flow-matching to shrink the gap between the learned distribution for a pretrained denoising diffusion model and the real data distribution.
  arXiv  Detail & Related papers  (2025-03-13T17:59:59Z)
• GE2E-AC: Generalized End-to-End Loss Training for Accent Classification [13.266765406714942]
  We propose a GE2E-AC, in which we train a model to extract accent embedding or AE of an input utterance. We experimentally show the effectiveness of the proposed GE2E-AC, compared to the baseline model trained with the conventional cross-entropy-based loss.
  arXiv  Detail & Related papers  (2024-07-19T04:44:16Z)
• DiffAug: A Diffuse-and-Denoise Augmentation for Training Robust Classifiers [6.131022957085439]
  We introduce DiffAug, a simple and efficient diffusion-based augmentation technique to train image classifiers. Applying DiffAug to a given example consists of one forward-diffusion step followed by one reverse-diffusion step.
  arXiv  Detail & Related papers  (2023-06-15T15:19:25Z)
• Zero-shot text-to-speech synthesis conditioned using self-supervised speech representation model [13.572330725278066]
  A novel point of the proposed method is the direct use of the SSL model to obtain embedding vectors from speech representations trained with a large amount of data. The disentangled embeddings will enable us to achieve better reproduction performance for unseen speakers and rhythm transfer conditioned by different speeches.
  arXiv  Detail & Related papers  (2023-04-24T10:15:58Z)
• Classifier-Free Diffusion Guidance [17.355749359987648]
  guidance is a recently introduced method of trade off mode coverage and sample fidelity in conditional diffusion models. We show that guidance can be indeed performed by a pure generative model without such a classifier. We combine the resulting conditional and unconditional score estimates to attain a trade-off between sample quality and diversity.
  arXiv  Detail & Related papers  (2022-07-26T01:42:07Z)
• Visualizing Classifier Adjacency Relations: A Case Study in Speaker Verification and Voice Anti-Spoofing [72.4445825335561]
  We propose a simple method to derive 2D representation from detection scores produced by an arbitrary set of binary classifiers. Based upon rank correlations, our method facilitates a visual comparison of classifiers with arbitrary scores. While the approach is fully versatile and can be applied to any detection task, we demonstrate the method using scores produced by automatic speaker verification and voice anti-spoofing systems.
  arXiv  Detail & Related papers  (2021-06-11T13:03:33Z)
• End-to-End Diarization for Variable Number of Speakers with Local-Global Networks and Discriminative Speaker Embeddings [66.50782702086575]
  We present an end-to-end deep network model that performs meeting diarization from single-channel audio recordings. The proposed system is designed to handle meetings with unknown numbers of speakers, using variable-number permutation-invariant cross-entropy based loss functions.
  arXiv  Detail & Related papers  (2021-05-05T14:55:29Z)
• Three-class Overlapped Speech Detection using a Convolutional Recurrent Neural Network [32.59704287230343]
  The proposed approach classifies into three classes: non-speech, single speaker speech, and overlapped speech. A convolutional recurrent neural network architecture is explored to benefit from both convolutional layer's capability to model local patterns and recurrent layer's ability to model sequential information. The proposed overlapped speech detection model establishes a state-of-the-art performance with a precision of 0.6648 and a recall of 0.3222 on the DIHARD II evaluation set.
  arXiv  Detail & Related papers  (2021-04-07T03:01:34Z)
• Learning and Evaluating Representations for Deep One-class Classification [59.095144932794646]
  We present a two-stage framework for deep one-class classification. We first learn self-supervised representations from one-class data, and then build one-class classifiers on learned representations. In experiments, we demonstrate state-of-the-art performance on visual domain one-class classification benchmarks.
  arXiv  Detail & Related papers  (2020-11-04T23:33:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.