End-to-End Training for Discrete Token LLM based TTS System

• URL: http://arxiv.org/abs/2606.09234v1
• Date: Mon, 08 Jun 2026 09:07:23 GMT
• Title: End-to-End Training for Discrete Token LLM based TTS System
• Abstract summary: We propose a fully end-to-end (E2E) optimization framework that unifies the training of the speech tokenizer, large language model, and flow-matching model.<n> Experimental results show that our framework consistently outperforms cascaded baselines.
• Score: 12.016326342176319
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent state-of-the-art (SOTA) text-to-speech (TTS) systems typically adopt a cascaded pipeline consisting of a speech tokenizer, an autoregressive large language model (LLM), and a diffusion based flow-matching (FM) model, with these components trained independently. In this paper, we propose a fully end-to-end (E2E) optimization framework that unifies the training of the speech tokenizer, LLM, FM model, and an additional reward model (RM). Specifically, we first jointly optimize the tokenizer using multi-task objectives derived from reconstruction for FM, next-token prediction for LLM, and multi recognition task for RM. This joint training encourages the discrete speech token space to capture acoustically and semantically salient information that is better tailored to TTS. We then further optimize the LLM using downstream reconstruction and recognition by FM and RM, which reduces inference-time mismatch and steers the LLM toward more preferred generations. Experimental results show that our E2E framework consistently outperforms cascaded baselines. On the Seed-TTS-Eval benchmark, our system achieves a word error rate (WER) of 0.78% and 1.56%, a new SOTA result with a 0.6B-parameter LLM and 0.5B-parameter FM model. These results validate that holistic E2E optimization is critical for improving discrete-token-based TTS systems with a much simpler training pipeline.

Related papers

• Meta-Transfer Learning for mmWave Beam Alignment [8.956048958779315]
  MTL-BA is a meta-transfer learning framework for beam alignment in millimeter-wave multiple-input single-output (MISO) systems.<n>We show that MTL-BA matches the accuracy and spectral efficiency of full fine-tuning across various SNR levels.
  arXiv  Detail & Related papers  (2026-07-01T12:24:48Z)
• Rethinking Entropy Allocation in LLM-based ASR: Understanding the Dynamics between Speech Encoders and LLMs [17.167595029948576]
  We introduce three metrics to characterize how training paradigms allocate entropy reduction between the speech encoder and the LLM.<n>We propose a principled multi-stage training strategy grounded in capability-boundary awareness, optimizing parameter efficiency and robustness hallucination.<n> Experiments on Mandarin and English benchmarks show that our method achieves competitive performance with state-of-the-art models using only 2.3B parameters.
  arXiv  Detail & Related papers  (2026-04-09T09:07:52Z)
• JoyAI-LLM Flash: Advancing Mid-Scale LLMs with Token Efficiency [56.46698214215968]
  JoyAI-LLM Flash is an efficient Mixture-of-Experts (MoE) language model designed to redefine the trade-off between strong performance and token efficiency.<n>JoyAI-LLM Flash is pretrained on a massive corpus of 20 trillion tokens and further optimized through a rigorous post-training pipeline.
  arXiv  Detail & Related papers  (2026-04-03T13:52:38Z)
• PROST-LLM: Progressively Enhancing the Speech-to-Speech Translation Capability in LLMs [13.891322931352649]
  We propose PROST-LLM (PROgressive Speech-to-speech Translation) to enhance the S2ST capabilities in Large Language Models.<n>First, we fine-tune the LLMs with the CVSS corpus, employing designed tri-task learning and chain of modality methods to boost the initial performance.<n>Finally, these preference pairs are used for preference optimization to enhance the model's S2ST capability further.
  arXiv  Detail & Related papers  (2026-01-23T10:16:13Z)
• Bridging the gap: A comparative exploration of Speech-LLM and end-to-end architecture for multilingual conversational ASR [16.090902570653803]
  We present an enhanced LLM-based ASR framework that combines fine-tuned Whisper and mHuBERT encoders with an LLM to enrich speech representations.<n>Our system achieves a CER/WER of 10.69%, ranking on par with the top-ranked Track 1 systems.
  arXiv  Detail & Related papers  (2026-01-04T10:08:53Z)
• Implicit Reward as the Bridge: A Unified View of SFT and DPO Connections [65.36449542323277]
  We present a unified theoretical framework bridgingSupervised Fine-Tuning (SFT) and preference learning in Large Language Model (LLM) post-training.<n>We propose a simple yet effective learning rate reduction approach that yields significant performance improvements.
  arXiv  Detail & Related papers  (2025-06-15T05:42:29Z)
• KIT's Low-resource Speech Translation Systems for IWSLT2025: System Enhancement with Synthetic Data and Model Regularization [64.1520245849231]
  This paper presents KIT's submissions to the IWSLT 2025 low-resource track.<n>We develop both cascaded systems, and end-to-end (E2E) Speech Translation systems.<n>Building upon pre-trained models, we fine-tune our systems with different strategies to utilize resources efficiently.
  arXiv  Detail & Related papers  (2025-05-26T08:38:02Z)
• Transducer-Llama: Integrating LLMs into Streamable Transducer-based Speech Recognition [26.79555533538622]
  This paper proposes a novel model architecture, Transducer-Llama, that integrates large language models (LLMs) into a Factorized Transducer (FT) model.<n>The proposed streaming Transducer-Llama approach gave a 17% relative WER reduction (WERR) over a strong FT baseline and a 32% WERR over an RNN-T baseline.
  arXiv  Detail & Related papers  (2024-12-21T03:35:49Z)
• Understanding Forgetting in LLM Supervised Fine-Tuning and Preference Learning - A Convex Optimization Perspective [55.66517396157806]
  The widely adopted approach in post-training popular open-source LLMs is to sequentially perform SFT and RLHF/DPO.<n>This is suboptimal in terms of SFT and RLHF/DPO trade-off.<n>We propose a practical joint post-training framework which has theoretical convergence guarantees and empirically outperforms sequential post-training framework.
  arXiv  Detail & Related papers  (2024-10-20T19:38:41Z)
• R-SFLLM: Jamming Resilient Framework for Split Federated Learning with Large Language Models [65.04475956174959]
  Split federated learning (SFL) is a compute-efficient paradigm in distributed machine learning (ML)<n>A significant challenge in SFL, particularly when deployed over wireless channels, is the susceptibility of transmitted model parameters to adversarial jamming.<n>This paper develops a physical layer framework for resilient SFL with large language models (LLMs) and vision language models (VLMs) over wireless networks.
  arXiv  Detail & Related papers  (2024-07-16T12:21:29Z)
• Bayesian Neural Network Language Modeling for Speech Recognition [59.681758762712754]
  State-of-the-art neural network language models (NNLMs) represented by long short term memory recurrent neural networks (LSTM-RNNs) and Transformers are becoming highly complex. In this paper, an overarching full Bayesian learning framework is proposed to account for the underlying uncertainty in LSTM-RNN and Transformer LMs.
  arXiv  Detail & Related papers  (2022-08-28T17:50:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.