Experience-Calibrated Contrastive Decoding for Mitigating Hallucinations in LM-Based Text-to-Speech

• URL: http://arxiv.org/abs/2608.00722v1
• Date: Sat, 01 Aug 2026 15:41:55 GMT
• Title: Experience-Calibrated Contrastive Decoding for Mitigating Hallucinations in LM-Based Text-to-Speech
• Abstract summary: Language model-based text-to-speech (LM-based TTS) remains vulnerable to speech hallucinations that deviate from the target text.<n>We present a conditional information view that distinguishes text-derived alignment information from experience information.<n>We propose Experience-Calibrated Contrastive Decoding (ECCD), a training-free method that strengthens alignment support while preserving useful experience information.
• Score: 25.75512357205199
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Language model-based text-to-speech (LM-based TTS) remains vulnerable to speech hallucinations that deviate from the target text. Existing mitigation mainly relies on architectural changes or additional training, while decoding-time control remains underexplored. We present a conditional information view that distinguishes text-derived alignment information from experience information supplied by acoustic context and learned speech regularities. We hypothesize that an important class of hallucinations begins when alignment support is insufficiently reflected in the selected token at a vulnerable transition. Using predictions from the same speech LM with and without text conditions, we propose Experience-Calibrated Contrastive Decoding (ECCD), a training-free method that strengthens alignment support while preserving useful experience information. ECCD preserves the original expert distribution, applies only positive alignment enhancement, and calibrates its strength using set-level experience compatibility. Across four models, ECCD reduces WER/CER by up to 55.6% in all SeedTTS-Eval settings and 24 of 25 multilingual CV3-Eval settings. A listening test yields a CMOS gain of $+0.644$ while retaining strong speaker similarity. Further analysis shows that alignment influence and decision-level gain vary within linguistic units and are lower at first-error boundaries than at matched correct boundaries. Overall, these extensive experiments and analyses identify conditional information control as a promising decoding-time direction for mitigating speech hallucination.

Related papers

• Hear2Act: Benchmarking When Prosody Should Change What an Assistant Does [27.909684871924004]
  We introduce Hear2Act, a unified evaluation protocol for text and spoken assistants.<n>We evaluate decisions under transcript, audio, and concern-state access.<n>We show that prosody matters when lexical evidence is insufficient.
  arXiv  Detail & Related papers  (2026-08-20T00:16:36Z)
• Inference-Time Steering for Cross-Lingual Factual Consistency in LLMs [51.56484100374058]
  Large Language Models (LLMs) demonstrate remarkable multilingual fluency, but their internal knowledge representations remain disproportionately biased toward high-resource languages.<n>We investigate whether these biases can be mitigated at inference time, forcing an English-prompted model to answer as if it were queried in target languages.<n>Experiments on Gemma 3 12B Instruct reveal persona prompting to be the strongest overall intervention, balancing efficacy, safety, and out-of-domain generalization.
  arXiv  Detail & Related papers  (2026-07-21T16:15:05Z)
• Transcription Policy as a Latent Variable: Activating Controllable Verbatim ASR with Word-Level Timing [8.921166277011347]
  We show that models already encode both styles; the challenge is controlled activation.<n>Using coverage-aware decoder task tokens trained on parallel verbatim/intended transcript pairs, we raise German disfluency F1 from 10% to 79% zero-shot.<n>Full English-only fine-tuning surpasses all baselines in verbatim accuracy, disfluency detection, and intended-mode quality across both languages.
  arXiv  Detail & Related papers  (2026-07-21T10:19:17Z)
• From Text Metrics to Model Internals: A Study of Whisper ASR Hallucination Detection [13.55239135877789]
  Hallucinations of ASR models pose risks in downstream applications.<n>This paper studies Whisper large v3 hallucination detection on real-speech human-annotated data.
  arXiv  Detail & Related papers  (2026-06-22T09:13:04Z)
• Few-Shot Contrastive Adaptation for Audio Abuse Detection in Low-Resource Indic Languages [3.5238606794194816]
  Abusive speech detection is becoming increasingly important as social media shifts towards voice-based interaction.<n>We investigate whether Contrastive Language-Audio Pre-training can support abusive speech detection directly from audio.
  arXiv  Detail & Related papers  (2026-04-10T08:23:03Z)
• ACT Now: Preempting LVLM Hallucinations via Adaptive Context Integration [71.21097024566285]
  Large-Language Models (LVLMs) frequently suffer from severe hallucination issues.<n>Existing mitigation strategies rely on linguistic, single-step states to enhance visual focus or suppress strong priors.<n>We propose Adaptive Context in VisionTegration (ACT), a training-free inference intervention method that mitigates hallucination through the adaptive integration of contextual information.
  arXiv  Detail & Related papers  (2026-04-01T14:49:50Z)
• Anatomy of the Modality Gap: Dissecting the Internal States of End-to-End Speech LLMs [15.914430317382077]
  We analyze how speech and text representations evolve layer-by-layer.<n>We find that speech representations exhibit a broad cross-layer alignment band, attributable to the redundant nature of speech.
  arXiv  Detail & Related papers  (2026-03-02T06:21:43Z)
• Eliminating stability hallucinations in llm-based tts models via attention guidance [54.69225348397135]
  This paper focuses on resolving stability hallucinations in Text-to-Speech (TTS) models by improving and leveraging the attention mechanism.<n>We proposed a metric termed the Optimal Alignment Score (OAS), which employs the Viterbi algorithm to evaluate text-speech alignment quality.<n>The proposed methods can effectively reduce the stability of CosyVoice2 without introducing additional negative effects.
  arXiv  Detail & Related papers  (2025-09-24T07:47:52Z)
• Representation Purification for End-to-End Speech Translation [16.967317436711113]
  Speech-to-text translation (ST) is a cross-modal task that involves converting spoken language into text in a different language.<n>We conceptualize speech representation as a combination of content-agnostic and content-relevant factors.
  arXiv  Detail & Related papers  (2024-12-05T15:50:44Z)
• Con-ReCall: Detecting Pre-training Data in LLMs via Contrastive Decoding [118.75567341513897]
  Existing methods typically analyze target text in isolation or solely with non-member contexts.<n>We propose Con-ReCall, a novel approach that leverages the asymmetric distributional shifts induced by member and non-member contexts.
  arXiv  Detail & Related papers  (2024-09-05T09:10:38Z)
• Prosody in Cascade and Direct Speech-to-Text Translation: a case study on Korean Wh-Phrases [79.07111754406841]
  This work proposes using contrastive evaluation to measure the ability of direct S2TT systems to disambiguate utterances where prosody plays a crucial role. Our results clearly demonstrate the value of direct translation systems over cascade translation models.
  arXiv  Detail & Related papers  (2024-02-01T14:46:35Z)
• Unlikelihood Tuning on Negative Samples Amazingly Improves Zero-Shot Translation [79.96416609433724]
  Zero-shot translation (ZST) aims to translate between unseen language pairs in training data. The common practice to guide the zero-shot language mapping during inference is to deliberately insert the source and target language IDs. Recent studies have shown that language IDs sometimes fail to navigate the ZST task, making them suffer from the off-target problem.
  arXiv  Detail & Related papers  (2023-09-28T17:02:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.