Effective Text Adaptation for LLM-based ASR through Soft Prompt Fine-Tuning

• URL: http://arxiv.org/abs/2412.06967v1
• Date: Mon, 09 Dec 2024 20:22:06 GMT
• Title: Effective Text Adaptation for LLM-based ASR through Soft Prompt Fine-Tuning
• Abstract summary: Large Language Models (LLM) has reformed the Automatic Speech Recognition (ASR)<n>Fine-tuning such ASR on text-only data without paired prompts may diminish the effectiveness of domain-specific knowledge.<n>We propose a two-step soft prompt fine-tuning strategy that enhances domain-specific text adaptation.
• Score: 12.676026149146772
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The advent of Large Language Models (LLM) has reformed the Automatic Speech Recognition (ASR). Prompting LLM with audio embeddings to generate transcriptions becomes the new state-of-the-art ASR. Despite LLMs being trained with an extensive amount of text corpora, high-quality domain-specific text data can still significantly enhance ASR performance on domain adaptation tasks. Although LLM-based ASR can naturally incorporate more text corpora by fine-tuning the LLM decoder, fine-tuning such ASR on text-only data without paired prompts may diminish the effectiveness of domain-specific knowledge. To mitigate this issue, we propose a two-step soft prompt fine-tuning strategy that enhances domain-specific text adaptation. Experimental results show that text adaptation with our proposed method achieved a relative up to 9% Word Error Rate (WER) reduction and up to 18% Entity Error Rate (EER) reduction on the target domain compared to the baseline ASR. Combining this with domain-specific Language Model (LM) fusion can further improve the EER by a relative 2-5%

Related papers

• Merging the Knowledge of LLMs for Automatic Speech Recognition [27.69035372273952]
  LM fusion methods incur additional computational costs due to LM inference.<n>We propose incorporating external LMs via model merging.<n>We show that our LM merging consistently improved the ASR performance in the target domains.
  arXiv  Detail & Related papers  (2026-09-14T15:35:22Z)
• Listen to the Latents: Self-Correcting Speech Recognition in Large Audio Language Models Through Hidden-State Interactions [120.64235456332916]
  Speech recognition systems increasingly integrate large language models (LLMs) to leverage their semantic knowledge.<n>We refine warm-adapted LLM-based ASR models by leveraging their own pre-adaptation base LLMs.<n>We propose Hybrid Search, a targeted correction strategy motivated by two observations.
  arXiv  Detail & Related papers  (2026-08-31T15:46:48Z)
• Speech LLMs are Contextual Reasoning Transcribers [29.079159231628026]
  Chain-of-thought ASR (CoT-ASR) constructs a reasoning chain that generates contextual analysis.<n>CoT-ASR naturally supports user-guided transcription.<n>It achieves a relative reduction of 8.7% in word error rate (WER) and 16.9% in entity error rate (EER)
  arXiv  Detail & Related papers  (2026-04-01T08:13:50Z)
• Adapting Text LLMs to Speech via Multimodal Depth Up-Scaling [52.02344262645619]
  We propose Multimodal Depth Upscaling, where new transformer layers are inserted into a frozen text LLM and only the added layers are trained on speech data.<n>Experiments with SmolLM2-360M and SmolLM2-1.7B on 48k hours of English Automatic Speech Recognition (ASR) data show that depth up-scaling achieves ASR comparable to full fine-tuning.<n>We further show that incorporating E-Branchformer, an architecture designed for speech recognition, as the inserted layers achieves ASR that matches or surpasses full fine-tuning on the larger model while reducing text degradation by over 75% with 60% fewer trainable parameters
  arXiv  Detail & Related papers  (2026-04-01T05:16:06Z)
• Text-only adaptation in LLM-based ASR through text denoising [14.200885240373509]
  Adapting automatic speech recognition systems to new domains using text-only data is a significant yet underexplored challenge.<n>We introduce a novel text-only adaptation method that emulates the audio projection task by treating it as a text denoising task.<n>Our solution is lightweight, requiring no architectural changes or additional parameters.
  arXiv  Detail & Related papers  (2026-01-28T10:18:23Z)
• SAFT: Structure-Aware Fine-Tuning of LLMs for AMR-to-Text Generation [50.277959544420455]
  SAFT is a structure-aware fine-tuning approach that injects graph topology into pretrained language models.<n>We compute direction-sensitive positional encodings from the magnetic Laplacian of transformed AMRs.<n> SAFT sets a new state-of-the-art on AMR 3.0 with a 3.5 BLEU improvement over baselines.
  arXiv  Detail & Related papers  (2025-07-15T18:12:57Z)
• Low-Resource Domain Adaptation for Speech LLMs via Text-Only Fine-Tuning [9.950088874229353]
  We propose a text-only fine-tuning strategy for Speech LLMs using unpaired target-domain text without requiring additional audio.<n>Experiments on LibriSpeech, SlideSpeech, and Medical datasets show that our method achieves competitive recognition performance.
  arXiv  Detail & Related papers  (2025-06-06T01:34:29Z)
• Delayed Fusion: Integrating Large Language Models into First-Pass Decoding in End-to-end Speech Recognition [17.376550014426623]
  This paper presents an efficient decoding approach for end-to-end automatic speech recognition (E2E-ASR) with large language models (LLMs) We propose "delayed fusion," which applies LLM scores to ASR hypotheses with a delay during decoding. We demonstrate that delayed fusion provides improved decoding speed and accuracy compared to shallow fusion and N-best rescoring.
  arXiv  Detail & Related papers  (2025-01-16T03:01:50Z)
• CTC-Assisted LLM-Based Contextual ASR [40.6542391788212]
  We propose a CTC-Assisted LLM-Based Contextual ASR model with an efficient filtering algorithm. Our model attains WER/B-WER of 1.27%/3.67% and 2.72%/8.02% on the Librispeech test-clean and test-other sets targeting on recognizing rare long-tail words.
  arXiv  Detail & Related papers  (2024-11-10T11:47:50Z)
• Bridging Speech and Text: Enhancing ASR with Pinyin-to-Character Pre-training in LLMs [20.97172337899685]
  We propose pre-training large language models (LLMs) on Pinyin embedding sequences to generate corresponding Chinese characters. This step enables the LLM to adapt to generating text from pronunciation features before encountering real speech data. In AISHELL-1 corpus, our approach yields a 9.5% relative improvement in ASR tasks compared to the baseline.
  arXiv  Detail & Related papers  (2024-09-24T12:06:31Z)
• Blending LLMs into Cascaded Speech Translation: KIT's Offline Speech Translation System for IWSLT 2024 [61.189875635090225]
  Large Language Models (LLMs) are currently under exploration for various tasks, including Automatic Speech Recognition (ASR), Machine Translation (MT), and even End-to-End Speech Translation (ST)
  arXiv  Detail & Related papers  (2024-06-24T16:38:17Z)
• One Token Can Help! Learning Scalable and Pluggable Virtual Tokens for Retrieval-Augmented Large Language Models [67.49462724595445]
  Retrieval-augmented generation (RAG) is a promising way to improve large language models (LLMs) We propose a novel method that involves learning scalable and pluggable virtual tokens for RAG.
  arXiv  Detail & Related papers  (2024-05-30T03:44:54Z)
• Data Augmentation for Text-based Person Retrieval Using Large Language Models [16.120524750964016]
  Text-based Person Retrieval (TPR) aims to retrieve person images that match the description given a text query. It is difficult to construct a large-scale, high-quality TPR dataset due to expensive annotation and privacy protection. This paper proposes an LLM-based Data Augmentation (LLM-DA) method for TPR.
  arXiv  Detail & Related papers  (2024-05-20T11:57:50Z)
• Unsupervised Information Refinement Training of Large Language Models for Retrieval-Augmented Generation [128.01050030936028]
  We propose an information refinement training method named InFO-RAG. InFO-RAG is low-cost and general across various tasks. It improves the performance of LLaMA2 by an average of 9.39% relative points.
  arXiv  Detail & Related papers  (2024-02-28T08:24:38Z)
• Improving Cross-Domain Low-Resource Text Generation through LLM Post-Editing: A Programmer-Interpreter Approach [50.400999859808984]
  Post-editing has proven effective in improving the quality of text generated by large language models (LLMs) We propose a neural programmer-interpreter approach that preserves the domain generalization ability of LLMs when editing their output. Experiments demonstrate that the programmer-interpreter significantly enhances GPT-3.5's performance in logical form-to-text conversion and low-resource machine translation.
  arXiv  Detail & Related papers  (2024-02-07T06:13:14Z)
• Correction Focused Language Model Training for Speech Recognition [14.246583065323192]
  We introduce a novel correction focused LM training approach which aims to prioritize ASR fallible words. The word-level ASR fallibility score is defined and shaped as a prior word distribution to guide the LM training. Compared with conventional LMs, correction focused training achieves up to relatively 5.5% word error rate (WER) reduction in sufficient text scenarios.
  arXiv  Detail & Related papers  (2023-10-17T05:10:39Z)
• Prompting Large Language Models for Zero-Shot Domain Adaptation in Speech Recognition [33.07184218085399]
  With only a domain-specific text prompt, we propose two zero-shot ASR domain adaptation methods using LLaMA. Experiments show that, with only one domain prompt, both methods can effectively reduce word error rates (WER) on out-of-domain TedLium-2 and SPGI datasets.
  arXiv  Detail & Related papers  (2023-06-28T08:29:00Z)
• Deliberation Model for On-Device Spoken Language Understanding [69.5587671262691]
  We propose a novel deliberation-based approach to end-to-end (E2E) spoken language understanding (SLU) We show that our approach can significantly reduce the degradation when moving from natural speech to synthetic speech training.
  arXiv  Detail & Related papers  (2022-04-04T23:48:01Z)
• Improving Readability for Automatic Speech Recognition Transcription [50.86019112545596]
  We propose a novel NLP task called ASR post-processing for readability (APR) APR aims to transform the noisy ASR output into a readable text for humans and downstream tasks while maintaining the semantic meaning of the speaker. We compare fine-tuned models based on several open-sourced and adapted pre-trained models with the traditional pipeline method.
  arXiv  Detail & Related papers  (2020-04-09T09:26:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.