Efficient ASR Training with Conversations that Never Happened

• URL: http://arxiv.org/abs/2606.03957v1
• Date: Tue, 02 Jun 2026 17:46:12 GMT
• Title: Efficient ASR Training with Conversations that Never Happened
• Abstract summary: We propose an augmentation pipeline that generates scenario-level dialogues with participant metadata.<n>We then map speaker attributes to TTS voice profiles, and assembles synthesized utterances into speaker-aware simulated conversations.
• Score: 1.0251581485267474
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Conversational ASR for lower-resource languages and niche domains is limited by the scarcity of domain-matched multi-speaker training data. We propose an augmentation pipeline that generates scenario-level dialogues with participant metadata, maps speaker attributes to TTS voice profiles, and assembles synthesized utterances into speaker-aware simulated conversations. We evaluated five LLM families under single-generator, fixed-budget mixture, and scale-up settings using the same FastConformer-Large training recipe for each one. We ran comprehensive evaluations on the Hungarian BEA-Dialogue benchmark corpus, with the method itself being applicable to any language given the resources for each component. The results show that synthetic conversations consistently improve speech recognition performance, but generator choice and data composition strongly affect the gains. Our largest training configuration, using only 67 hours of real conversations and 636 hours of simulated data, achieves better performance on the evaluation benchmark than a zero-shot model trained on 2700 hours of Hungarian speech. These findings indicate that LLM-generated conversational data synthesized with TTS is a practical complement to real conversational corpora for speech model training.

Related papers

• Scaling Conversational Hungarian ASR: The BEA-Dialogue+ Corpus [1.3432835891025114]
  We introduce BEA-Dialogue+, an expanded version of the BEA-Dialogue corpus that relaxes the split criterion for experimenters and dialogue partners.<n>This results in 200 hours of transcribed natural conversations.<n>We evaluate several Whisper- and FastConformer-based models on both corpus versions.
  arXiv  Detail & Related papers  (2026-05-29T16:01:25Z)
• MUSCAT: MUltilingual, SCientific ConversATion Benchmark [60.87925076316812]
  The goal of multilingual speech technology is to facilitate seamless communication between individuals speaking different languages.<n>To create this experience, speech technology needs to address several challenges: Handling mixed multilingual input, specific vocabulary, and code-switching.<n>We propose a new benchmark to evaluate current Automatic Speech Recognition (ASR) systems, whether they are able to handle these challenges.
  arXiv  Detail & Related papers  (2026-04-17T10:39:01Z)
• Speaker-Aware Simulation Improves Conversational Speech Recognition [1.0251581485267474]
  We adapt and implement the SASC framework for Hungarian conversational ASR.<n>We propose C-SASC, an extended variant that incorporates pause modeling conditioned on utterance duration.<n>We generate synthetic Hungarian dialogues from the BEA-Large corpus and combine them with real conversational data for ASR training.
  arXiv  Detail & Related papers  (2026-02-04T17:12:09Z)
• PRiSM: Benchmarking Phone Realization in Speech Models [70.82595415252682]
  Phone recognition (PR) serves as the atomic interface for language-agnostic modeling for cross-lingual speech processing and phonetic analysis.<n>We introduce PRiSM, the first open-source benchmark designed to expose blind spots in phonetic perception.
  arXiv  Detail & Related papers  (2026-01-20T15:00:36Z)
• Bi-directional Context-Enhanced Speech Large Language Models for Multilingual Conversational ASR [23.285609467633865]
  This paper introduces the integration of language-specific bi-directional context into a speech large language model (SLLM) to improve multilingual continuous conversational automatic speech recognition (ASR)<n>We propose a character-level contextual masking strategy during training, which randomly removes portions of the context to enhance robustness and better emulate the flawed transcriptions that may occur during inference.
  arXiv  Detail & Related papers  (2025-06-16T12:03:23Z)
• Leveraging LLM and Self-Supervised Training Models for Speech Recognition in Chinese Dialects: A Comparative Analysis [4.774607166378613]
  Self-supervised pre-training, combined with large language models (LLM), can effectively enhance ASR performance in low-resource scenarios.<n>We pre-train a Data2vec2 model on 300,000 hours of unlabeled dialect and accented speech data and do alignment training on a supervised dataset of 40,000 hours.
  arXiv  Detail & Related papers  (2025-05-27T12:50:55Z)
• End-to-End Speech Recognition Contextualization with Large Language Models [25.198480789044346]
  We introduce a novel method for contextualizing speech recognition models incorporating Large Language Models (LLMs) We provide audio features, along with optional text tokens for context, to train the system to complete transcriptions in a decoder-only fashion. Our empirical results demonstrate a significant improvement in performance, with a 6% WER reduction when additional textual context is provided.
  arXiv  Detail & Related papers  (2023-09-19T20:28:57Z)
• ComSL: A Composite Speech-Language Model for End-to-End Speech-to-Text Translation [79.66359274050885]
  We present ComSL, a speech-language model built atop a composite architecture of public pretrained speech-only and language-only models. Our approach has demonstrated effectiveness in end-to-end speech-to-text translation tasks.
  arXiv  Detail & Related papers  (2023-05-24T07:42:15Z)
• WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing [102.45426364965887]
  We propose a new pre-trained model, WavLM, to solve full-stack downstream speech tasks. WavLM is built based on the HuBERT framework, with an emphasis on both spoken content modeling and speaker identity preservation. We scale up the training dataset from 60k hours to 94k hours of public audio data, and optimize its training procedure for better representation extraction.
  arXiv  Detail & Related papers  (2021-10-26T17:55:19Z)
• Unsupervised Cross-lingual Representation Learning for Speech Recognition [63.85924123692923]
  XLSR learns cross-lingual speech representations by pretraining a single model from the raw waveform of speech in multiple languages. We build on wav2vec 2.0 which is trained by solving a contrastive task over masked latent speech representations. Experiments show that cross-lingual pretraining significantly outperforms monolingual pretraining.
  arXiv  Detail & Related papers  (2020-06-24T18:25:05Z)
• Semi-supervised Learning for Multi-speaker Text-to-speech Synthesis Using Discrete Speech Representation [125.59372403631006]
  We propose a semi-supervised learning approach for multi-speaker text-to-speech (TTS) A multi-speaker TTS model can learn from the untranscribed audio via the proposed encoder-decoder framework with discrete speech representation. We found the model can benefit from the proposed semi-supervised learning approach even when part of the unpaired speech data is noisy.
  arXiv  Detail & Related papers  (2020-05-16T15:47:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.