Eliminating stability hallucinations in llm-based tts models via attention guidance

• URL: http://arxiv.org/abs/2509.19852v1
• Date: Wed, 24 Sep 2025 07:47:52 GMT
• Title: Eliminating stability hallucinations in llm-based tts models via attention guidance
• Abstract summary: This paper focuses on resolving stability hallucinations in Text-to-Speech (TTS) models by improving and leveraging the attention mechanism.<n>We proposed a metric termed the Optimal Alignment Score (OAS), which employs the Viterbi algorithm to evaluate text-speech alignment quality.<n>The proposed methods can effectively reduce the stability of CosyVoice2 without introducing additional negative effects.
• Score: 54.69225348397135
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper focuses on resolving stability hallucinations (e.g., repetitive or omitted speech) in LLM-based Text-to-Speech (TTS) models by improving and leveraging the attention mechanism. First, we analyzed the alignment mechanism between text tokens and speech tokens in LLMs. We then proposed a metric termed the Optimal Alignment Score (OAS), which employs the Viterbi algorithm to evaluate text-speech alignment quality. Subsequently, OAS was integrated into the training of CosyVoice2 to assist LLMs in learning continuous, stable alignment. Additionally, the pre-trained attention value is employed to guide the training of the student CosyVoice2 via chain-of-thought (CoT), which further reduces stability hallucinations in synthesized speech. Experiments on the Seed-TTS-Eval and CV3-Eval test sets demonstrate that the proposed methods can effectively reduce the stability hallucinations of CosyVoice2 without introducing additional negative effects. The appendix is available at https://wsmzzz.github.io/llm_attn.

Related papers

• Robust Summarization of Doctor-Patient Conversations: TalTech Systems for the Beyond Transcription Challenge [6.193919591996934]
  This paper describes TalTech's submissions to the Beyond Transcription Challenge (BeTraC)<n>The challenge requires generating SOAP notes directly from long doctor-patient conversation recordings, without intermediate transcription.<n>We adapted Voxtral Mini and Voxtral Small for long-audio robustness, with LoRA supervised fine-tuning followed by DAPO reinforcement learning.<n>We find that fine-tuning on text transcripts transfers well to speech input and appears to improve robustness on out-of-domain real recordings.
  arXiv  Detail & Related papers  (2026-07-19T12:49:35Z)
• From Text Metrics to Model Internals: A Study of Whisper ASR Hallucination Detection [13.55239135877789]
  Hallucinations of ASR models pose risks in downstream applications.<n>This paper studies Whisper large v3 hallucination detection on real-speech human-annotated data.
  arXiv  Detail & Related papers  (2026-06-22T09:13:04Z)
• Hearing Between the Lines: Unlocking the Reasoning Power of LLMs for Speech Evaluation [19.92868268408954]
  Large Language Model (LLM) judges exhibit strong reasoning capabilities but are limited to textual content.<n>We propose TRACE, a novel framework that enables LLM judges to reason over audio cues to achieve cost-efficient and human-aligned S2S evaluation.<n>We will release the HCoT annotations and the TRACE framework to enable scalable and human-aligned S2S evaluation.
  arXiv  Detail & Related papers  (2026-01-20T08:57:02Z)
• S2SBench: A Benchmark for Quantifying Intelligence Degradation in Speech-to-Speech Large Language Models [14.060679420379516]
  End-to-end speech large language models ((LLMs)) extend the capabilities of text-based models to directly process and generate audio tokens.<n>This often leads to a decline in reasoning and generation performance compared to text input.<n>We propose S2SBench, a benchmark designed to quantify performance degradation in Speech LLMs.
  arXiv  Detail & Related papers  (2025-05-20T14:42:20Z)
• GOAT-TTS: Expressive and Realistic Speech Generation via A Dual-Branch LLM [42.93855899824886]
  We propose a text-to-speech generation approach optimized via a novel dual-branch ArchiTecture (GOAT-TTS)<n>GOAT-TTS combines a speech encoder and projector to capture continuous acoustic embeddings, enabling bidirectional correlation between paralinguistic features (language, timbre, emotion) and semantic text representations without transcript dependency.<n> Experimental results demonstrate that our GOAT-TTS achieves performance comparable to state-of-the-art TTS models.
  arXiv  Detail & Related papers  (2025-04-15T01:44:56Z)
• MegaTTS 3: Sparse Alignment Enhanced Latent Diffusion Transformer for Zero-Shot Speech Synthesis [56.25862714128288]
  This paper introduces textitMegaTTS 3, a zero-shot text-to-speech (TTS) system featuring an innovative sparse alignment algorithm.<n>Specifically, we provide sparse alignment boundaries to MegaTTS 3 to reduce the difficulty of alignment without limiting the search space.<n>Experiments demonstrate that MegaTTS 3 achieves state-of-the-art zero-shot TTS speech quality and supports highly flexible control over accent intensity.
  arXiv  Detail & Related papers  (2025-02-26T08:22:00Z)
• Speech collage: code-switched audio generation by collaging monolingual corpora [50.356820349870986]
  Speech Collage is a method that synthesizes CS data from monolingual corpora by splicing audio segments. We investigate the impact of generated data on speech recognition in two scenarios.
  arXiv  Detail & Related papers  (2023-09-27T14:17:53Z)
• High-Quality Automatic Voice Over with Accurate Alignment: Supervision through Self-Supervised Discrete Speech Units [69.06657692891447]
  We propose a novel AVO method leveraging the learning objective of self-supervised discrete speech unit prediction. Experimental results show that our proposed method achieves remarkable lip-speech synchronization and high speech quality.
  arXiv  Detail & Related papers  (2023-06-29T15:02:22Z)
• Weakly-supervised forced alignment of disfluent speech using phoneme-level modeling [10.283092375534311]
  We propose a simple and effective modification of alignment graph construction using weighted Finite State Transducers. The proposed weakly-supervised approach alleviates the need for verbatim transcription of speech disfluencies for forced alignment. Our evaluation on a corrupted version of the TIMIT test set and the UCLASS dataset shows significant improvements.
  arXiv  Detail & Related papers  (2023-05-30T09:57:36Z)
• Audio-Visual Efficient Conformer for Robust Speech Recognition [91.3755431537592]
  We propose to improve the noise of the recently proposed Efficient Conformer Connectionist Temporal Classification architecture by processing both audio and visual modalities. Our experiments show that using audio and visual modalities allows to better recognize speech in the presence of environmental noise and significantly accelerate training, reaching lower WER with 4 times less training steps.
  arXiv  Detail & Related papers  (2023-01-04T05:36:56Z)
• Improved Beam Search for Hallucination Mitigation in Abstractive Summarization [1.2328446298523066]
  In this paper, we investigate the use of the Natural Language Inference (NLI) entailment metric to detect and prevent hallucinations in summary generation. We propose an NLI-assisted beam re-ranking mechanism by computing entailment probability scores between the input context and summarization model-generated beams. Our proposed algorithm significantly outperforms vanilla beam decoding on XSum and CNN/DM datasets.
  arXiv  Detail & Related papers  (2022-12-06T02:33:47Z)
• Supervision-Guided Codebooks for Masked Prediction in Speech Pre-training [102.14558233502514]
  Masked prediction pre-training has seen remarkable progress in self-supervised learning (SSL) for speech recognition. We propose two supervision-guided codebook generation approaches to improve automatic speech recognition (ASR) performance.
  arXiv  Detail & Related papers  (2022-06-21T06:08:30Z)
• Continuous Speech Separation with Conformer [60.938212082732775]
  We use transformer and conformer in lieu of recurrent neural networks in the separation system. We believe capturing global information with the self-attention based method is crucial for the speech separation.
  arXiv  Detail & Related papers  (2020-08-13T09:36:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.