LI-TTA: Language Informed Test-Time Adaptation for Automatic Speech Recognition

• URL: http://arxiv.org/abs/2408.05769v1
• Date: Sun, 11 Aug 2024 13:19:27 GMT
• Title: LI-TTA: Language Informed Test-Time Adaptation for Automatic Speech Recognition
• Authors: Eunseop Yoon, Hee Suk Yoon, John Harvill, Mark Hasegawa-Johnson, Chang D. Yoo,
• Abstract summary: Test-Time Adaptation (TTA) has emerged as a crucial solution to the domain shift challenge. We propose Language Informed Test-Time Adaptation (LI-TTA), which incorporates linguistic insights during TTA for ASR.
• Score: 43.19328760778868
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Test-Time Adaptation (TTA) has emerged as a crucial solution to the domain shift challenge, wherein the target environment diverges from the original training environment. A prime exemplification is TTA for Automatic Speech Recognition (ASR), which enhances model performance by leveraging output prediction entropy minimization as a self-supervision signal. However, a key limitation of this self-supervision lies in its primary focus on acoustic features, with minimal attention to the linguistic properties of the input. To address this gap, we propose Language Informed Test-Time Adaptation (LI-TTA), which incorporates linguistic insights during TTA for ASR. LI-TTA integrates corrections from an external language model to merge linguistic with acoustic information by minimizing the CTC loss from the correction alongside the standard TTA loss. With extensive experiments, we show that LI-TTA effectively improves the performance of TTA for ASR in various distribution shift situations.

Related papers

• SUTA-LM: Bridging Test-Time Adaptation and Language Model Rescoring for Robust ASR [58.31068047426522]
  Test-Time Adaptation (TTA) aims to mitigate by adjusting models during inference.<n>Recent work explores combining TTA with external language models, using techniques like beam search rescoring or generative error correction.<n>We propose SUTA-LM, a simple yet effective extension of SUTA, with language model rescoring.<n> Experiments on 18 diverse ASR datasets show that SUTA-LM achieves robust results across a wide range of domains.
  arXiv  Detail & Related papers  (2025-06-10T02:50:20Z)
• HENT-SRT: Hierarchical Efficient Neural Transducer with Self-Distillation for Joint Speech Recognition and Translation [19.997594859651233]
  HENT-SRT is a novel framework that factorizes ASR and translation tasks to better handle reordering.<n>We improve computational efficiency by incorporating best practices from ASR transducers.<n>Our approach is evaluated on three conversational datasets Arabic, Spanish, and Mandarin.
  arXiv  Detail & Related papers  (2025-06-02T18:37:50Z)
• Temporal Order Preserved Optimal Transport-based Cross-modal Knowledge Transfer Learning for ASR [36.250914527327005]
  Transferring linguistic knowledge from a pretrained language model to an acoustic model has been shown to greatly improve the performance of automatic speech recognition. We propose a Temporal Order Preserved OT (TOT)-based Cross-modal Alignment and Knowledge Transfer (CAKT) for ASR.
  arXiv  Detail & Related papers  (2024-09-03T19:11:15Z)
• An Initial Investigation of Language Adaptation for TTS Systems under Low-resource Scenarios [76.11409260727459]
  This paper explores the language adaptation capability of ZMM-TTS, a recent SSL-based multilingual TTS system. We demonstrate that the similarity in phonetics between the pre-training and target languages, as well as the language category, affects the target language's adaptation performance.
  arXiv  Detail & Related papers  (2024-06-13T08:16:52Z)
• Self-supervised Adaptive Pre-training of Multilingual Speech Models for Language and Dialect Identification [19.893213508284813]
  Self-supervised adaptive pre-training is proposed to adapt the pre-trained model to the target domain and languages of the downstream task. We show that SAPT improves XLSR performance on the FLEURS benchmark with substantial gains up to 40.1% for under-represented languages.
  arXiv  Detail & Related papers  (2023-12-12T14:58:08Z)
• Parameter-Efficient Learning for Text-to-Speech Accent Adaptation [58.356667204518985]
  This paper presents a parameter-efficient learning (PEL) to develop a low-resource accent adaptation for text-to-speech (TTS) A resource-efficient adaptation from a frozen pre-trained TTS model is developed by using only 1.2% to 0.8% of original trainable parameters. Experiment results show that the proposed methods can achieve competitive naturalness with parameter-efficient decoder fine-tuning.
  arXiv  Detail & Related papers  (2023-05-18T22:02:59Z)
• Code-Switching Text Generation and Injection in Mandarin-English ASR [57.57570417273262]
  We investigate text generation and injection for improving the performance of an industry commonly-used streaming model, Transformer-Transducer (T-T) We first propose a strategy to generate code-switching text data and then investigate injecting generated text into T-T model explicitly by Text-To-Speech (TTS) conversion or implicitly by tying speech and text latent spaces. Experimental results on the T-T model trained with a dataset containing 1,800 hours of real Mandarin-English code-switched speech show that our approaches to inject generated code-switching text significantly boost the performance of T-T models.
  arXiv  Detail & Related papers  (2023-03-20T09:13:27Z)
• Using External Off-Policy Speech-To-Text Mappings in Contextual End-To-End Automated Speech Recognition [19.489794740679024]
  We investigate the potential of leveraging external knowledge, particularly through off-policy key-value stores generated with text-to-speech methods. In our approach, audio embeddings captured from text-to-speech, along with semantic text embeddings, are used to bias ASR. Experiments on LibiriSpeech and in-house voice assistant/search datasets show that the proposed approach can reduce domain adaptation time by up to 1K GPU-hours.
  arXiv  Detail & Related papers  (2023-01-06T22:32:50Z)
• Cross-Utterance Conditioned VAE for Non-Autoregressive Text-to-Speech [27.84124625934247]
  Cross-utterance conditional VAE is proposed to estimate a posterior probability distribution of the latent prosody features for each phoneme. CUC-VAE allows sampling from an utterance-specific prior distribution conditioned on cross-utterance information. Experimental results on LJ-Speech and LibriTTS data show that the proposed CUC-VAE TTS system improves naturalness and prosody diversity with clear margins.
  arXiv  Detail & Related papers  (2022-05-09T08:39:53Z)
• Listen, Adapt, Better WER: Source-free Single-utterance Test-time Adaptation for Automatic Speech Recognition [65.84978547406753]
  Test-time Adaptation aims to adapt the model trained on source domains to yield better predictions for test samples. Single-Utterance Test-time Adaptation (SUTA) is the first TTA study in speech area to our best knowledge.
  arXiv  Detail & Related papers  (2022-03-27T06:38:39Z)
• ATCSpeechNet: A multilingual end-to-end speech recognition framework for air traffic control systems [15.527854608553824]
  ATCSpeechNet is proposed to tackle the issue of translating communication speech into human-readable text in air traffic control systems. An end-to-end paradigm is developed to convert speech waveform into text directly, without any feature engineering or lexicon. Experimental results on the ATCSpeech corpus demonstrate that the proposed approach achieves a high performance with a very small labeled corpus.
  arXiv  Detail & Related papers  (2021-02-17T02:27:09Z)
• Joint Contextual Modeling for ASR Correction and Language Understanding [60.230013453699975]
  We propose multi-task neural approaches to perform contextual language correction on ASR outputs jointly with language understanding (LU) We show that the error rates of off the shelf ASR and following LU systems can be reduced significantly by 14% relative with joint models trained using small amounts of in-domain data.
  arXiv  Detail & Related papers  (2020-01-28T22:09:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.