ILT-Iterative LoRA Training through Focus-Feedback-Fix for Multilingual Speech Recognition

• URL: http://arxiv.org/abs/2507.08477v1
• Date: Fri, 11 Jul 2025 10:38:51 GMT
• Title: ILT-Iterative LoRA Training through Focus-Feedback-Fix for Multilingual Speech Recognition
• Authors: Qingliang Meng, Hao Wu, Wei Liang, Wei Xu, Qing Zhao,
• Abstract summary: This work proposes an innovative training paradigm Iterative LoRA Training (ILT) in combination with an Iterative Pseudo Labeling strategy.<n>Based on Whisper-large-v3 and Qwen2-Audio, we conduct systematic experiments using a three-stage training process: Focus Training, Feed Back Training, and Fix Training.<n>The MegaAIS research team applied this technique in the Interspeech 2025 Multilingual Conversational Speech Language Modeling Challenge (MLC-SLM), achieving 4th in Track 1 and 1st place in Track 2.
• Score: 14.909498057247648
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The deep integration of large language models and automatic speech recognition systems has become a promising research direction with high practical value. To address the overfitting issue commonly observed in Low-Rank Adaptation (LoRA) during the supervised fine-tuning (SFT) stage, this work proposes an innovative training paradigm Iterative LoRA Training (ILT) in combination with an Iterative Pseudo Labeling strategy, effectively enhancing the theoretical upper bound of model performance. Based on Whisper-large-v3 and Qwen2-Audio, we conduct systematic experiments using a three-stage training process: Focus Training, Feed Back Training, and Fix Training. Experimental results demonstrate the effectiveness of the proposed method. Furthermore, the MegaAIS research team applied this technique in the Interspeech 2025 Multilingual Conversational Speech Language Modeling Challenge (MLC-SLM), achieving 4th in Track 1 (Multilingual ASR Task) and 1st place in Track 2 (Speech Separation and Recognition Task), showcasing the practical feasibility and strong application potential of our approach.

Related papers

• The Eloquence team submission for task 1 of MLC-SLM challenge [12.442720957990533]
  We present our studies and experiments carried out for the task 1 of the Challenge and Workshop on Multilingual Conversational Speech Language Model (MLC-SLM)<n>This task focuses on advancing multilingual conversational speech recognition through the development of speech language models architectures.
  arXiv  Detail & Related papers  (2025-07-25T14:23:49Z)
• Triple X: A LLM-Based Multilingual Speech Recognition System for the INTERSPEECH2025 MLC-SLM Challenge [24.966911190845817]
  This paper describes our Triple X speech recognition system submitted to Task 1 of the Multi-Lingual Conversational Speech Language Modeling (MLC-SLM) Challenge.<n>Our work focuses on optimizing speech recognition accuracy in multilingual conversational scenarios through an innovative encoder-adapter-LLM architecture.
  arXiv  Detail & Related papers  (2025-07-23T07:48:33Z)
• Seewo's Submission to MLC-SLM: Lessons learned from Speech Reasoning Language Models [4.917936997225074]
  Seewo's systems for both tracks of the Multilingual Conversational Speech Language Model Challenge (MLC-SLM)<n>We introduce a multi-stage training pipeline that explicitly enhances reasoning and self-correction in speech language models for ASR.
  arXiv  Detail & Related papers  (2025-06-16T09:42:05Z)
• Towards Robust Overlapping Speech Detection: A Speaker-Aware Progressive Approach Using WavLM [53.17360668423001]
  Overlapping Speech Detection (OSD) aims to identify regions where multiple speakers overlap in a conversation.<n>This work proposes a speaker-aware progressive OSD model that leverages a progressive training strategy to enhance the correlation between subtasks.<n> Experimental results show that the proposed method achieves state-of-the-art performance, with an F1 score of 82.76% on the AMI test set.
  arXiv  Detail & Related papers  (2025-05-29T07:47:48Z)
• Analyzing Mitigation Strategies for Catastrophic Forgetting in End-to-End Training of Spoken Language Models [79.90523648823522]
  Multi-stage continual learning can lead to catastrophic forgetting.<n>This paper evaluates three mitigation strategies-model merging, discounting the LoRA scaling factor, and experience replay.<n>Results show that experience replay is the most effective, with further gains achieved by combining it with other methods.
  arXiv  Detail & Related papers  (2025-05-23T05:50:14Z)
• Low-Rank and Sparse Model Merging for Multi-Lingual Speech Recognition and Translation [7.437128866333448]
  We introduce LoRS-Merging, a novel technique designed to efficiently integrate models trained on different languages or tasks.<n>LoRS-Merging combines low-rank and sparse pruning to retain essential structures while eliminating redundant parameters.<n> Experimental results across 10 languages demonstrate that LoRS-Merging significantly outperforms multi-lingual multi-task training.
  arXiv  Detail & Related papers  (2025-02-24T18:06:57Z)
• Scalable Language Model with Generalized Continual Learning [58.700439919096155]
  The Joint Adaptive Re-ization (JARe) is integrated with Dynamic Task-related Knowledge Retrieval (DTKR) to enable adaptive adjustment of language models based on specific downstream tasks. Our method demonstrates state-of-the-art performance on diverse backbones and benchmarks, achieving effective continual learning in both full-set and few-shot scenarios with minimal forgetting.
  arXiv  Detail & Related papers  (2024-04-11T04:22:15Z)
• WavLLM: Towards Robust and Adaptive Speech Large Language Model [93.0773293897888]
  We introduce WavLLM, a robust and adaptive speech large language model with dual encoders, and a prompt-aware LoRA weight adapter. We validate the proposed model on universal speech benchmarks including tasks such as ASR, ST, SV, ER, and also apply it to specialized datasets like Gaokao English listening comprehension set for SQA, and speech Chain-of-Thought (CoT) evaluation set.
  arXiv  Detail & Related papers  (2024-03-31T12:01:32Z)
• UniSpeech-SAT: Universal Speech Representation Learning with Speaker Aware Pre-Training [72.004873454347]
  Two methods are introduced for enhancing the unsupervised speaker information extraction. Experiment results on SUPERB benchmark show that the proposed system achieves state-of-the-art performance. We scale up training dataset to 94 thousand hours public audio data and achieve further performance improvement.
  arXiv  Detail & Related papers  (2021-10-12T05:43:30Z)
• Structural Pre-training for Dialogue Comprehension [51.215629336320305]
  We present SPIDER, Structural Pre-traIned DialoguE Reader, to capture dialogue exclusive features. To simulate the dialogue-like features, we propose two training objectives in addition to the original LM objectives. Experimental results on widely used dialogue benchmarks verify the effectiveness of the newly introduced self-supervised tasks.
  arXiv  Detail & Related papers  (2021-05-23T15:16:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.