Proficiency-Aware Adaptation and Data Augmentation for Robust L2 ASR

• URL: http://arxiv.org/abs/2510.10738v1
• Date: Sun, 12 Oct 2025 18:20:58 GMT
• Title: Proficiency-Aware Adaptation and Data Augmentation for Robust L2 ASR
• Abstract summary: General-purpose ASR underperforms for atypical speakers, such as L2 learners, reinforcing bias and limiting use in education and accessibility.<n>We show that naive fine-tuning of Whisper reduces average WER but simultaneously widens disparities and disproportionately harms lower-level learners.<n>We propose proficiency-aware multitask learning, jointly optimizing ASR with proficiency classification, and (ii) targeted augmentation, applying spectrogram masking to low-proficiency speech to counter imbalance.
• Score: 1.9051696778411429
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: General-purpose ASR underperforms for atypical speakers, such as L2 learners, reinforcing bias and limiting use in education and accessibility. Using the CEFR-graded Speak and Improve corpus, we show that naive fine-tuning of Whisper reduces average WER but simultaneously widens disparities and disproportionately harms lower-level learners. To address this, we propose two strategies: (i) proficiency-aware multitask learning, jointly optimizing ASR with proficiency classification, and (ii) targeted augmentation, applying spectrogram masking to low-proficiency speech to counter imbalance. These approaches reduce WER by up to 29.4 percent (relative) and insertion/deletion errors by as much as 58.6 percent (relative). Crucially, despite the severe imbalance of the dataset reflecting real-world distributions, both strategies consistently narrow proficiency gaps, advancing equitable ASR for L2 learners.

Related papers

• Towards Personalized Federated Learning for Dysarthric Speech Recognition [49.291957451546914]
  This paper explores two aggregation strategies to achieve personalization, including the parameter-based averaging strategy and the embedding-based averaging strategy.<n> Experiments on UASpeech and TORGO show that the proposed methods outperform the baseline regularized FedAvg by statistically significant WER reductions of up to 0.99% absolute (3.15% relative) on UASpeech and 0.56% absolute (4.73% relative) on TORGO, respectively.
  arXiv  Detail & Related papers  (2026-06-11T12:10:57Z)
• Towards Fair ASR For Second Language Speakers Using Fairness Prompted Finetuning [37.91306016310542]
  We address the challenge of building fair English ASR systems for second-language speakers.<n>Our analysis of widely used ASR models, Whisper and Seamless-M4T, reveals large fluctuations in word error rate (WER) across 26 accent groups.<n>To mitigate this, we propose fairness-prompted finetuning with lightweight adapters.
  arXiv  Detail & Related papers  (2025-10-21T07:45:21Z)
• Wisdom is Knowing What not to Say: Hallucination-Free LLMs Unlearning via Attention Shifting [11.725875396424927]
  We introduce Attention-Shifting (AS) framework for selective unlearning.<n>AS is driven by two design objectives: (1) context-preserving suppression that attenuates attention to fact-bearing tokens without disrupting LLMs' linguistic structure; and (2) hallucination-resistant response shaping that discourages fabricated completions when queried about unlearning content.<n> Experimental results show that AS improves performance over the state-of-the-art unlearning methods, achieving up to 15% higher accuracy on the ToFU benchmark and 10% on the TDEC benchmark, while maintaining competitive hallucination-free unlearning effectiveness.
  arXiv  Detail & Related papers  (2025-10-20T06:50:03Z)
• Bridging ASR and LLMs for Dysarthric Speech Recognition: Benchmarking Self-Supervised and Generative Approaches [0.0]
  Speech Recognition (ASR) due to phoneme distortions and high variability.<n>Self-supervised ASR models like Wav2Vec, HuBERT, and Whisper have shown promise, but their effectiveness in dysarthric speech remains unclear.<n>This study systematically benchmarks these models with different decoding strategies, including CTC, seq2seq, and LLM-enhanced decoding.
  arXiv  Detail & Related papers  (2025-08-11T14:31:20Z)
• T2I-Eval-R1: Reinforcement Learning-Driven Reasoning for Interpretable Text-to-Image Evaluation [60.620408007636016]
  We propose T2I-Eval-R1, a novel reinforcement learning framework that trains open-source MLLMs using only coarse-grained quality scores.<n>Our approach integrates Group Relative Policy Optimization into the instruction-tuning process, enabling models to generate both scalar scores and interpretable reasoning chains.
  arXiv  Detail & Related papers  (2025-05-23T13:44:59Z)
• Inclusive ASR for Disfluent Speech: Cascaded Large-Scale Self-Supervised Learning with Targeted Fine-Tuning and Data Augmentation [0.0]
  A critical barrier to progress is the scarcity of large, annotated disfluent speech datasets. We present an inclusive ASR design approach, leveraging self-supervised learning on standard speech followed by targeted fine-tuning and data augmentation. Results show that fine-tuning wav2vec 2.0 with even a relatively small, labeled dataset, alongside data augmentation, can significantly reduce word error rates for disfluent speech.
  arXiv  Detail & Related papers  (2024-06-14T16:56:40Z)
• Joint Unsupervised and Supervised Training for Automatic Speech Recognition via Bilevel Optimization [73.98386682604122]
  We present a novel bilevel optimization-based training approach to training acoustic models for automatic speech recognition (ASR) tasks that we term bi-level joint unsupervised and supervised training (BL-JUST). BL-JUST employs a lower and upper level optimization with an unsupervised loss and a supervised loss respectively, leveraging recent advances in penalty-based bilevel optimization to solve this challenging ASR problem with affordable complexity and rigorous convergence guarantees.
  arXiv  Detail & Related papers  (2024-01-13T05:01:47Z)
• ML-LMCL: Mutual Learning and Large-Margin Contrastive Learning for Improving ASR Robustness in Spoken Language Understanding [55.39105863825107]
  We propose Mutual Learning and Large-Margin Contrastive Learning (ML-LMCL) to improve automatic speech recognition (ASR) robustness. In fine-tuning, we apply mutual learning and train two SLU models on the manual transcripts and the ASR transcripts, respectively. Experiments on three datasets show that ML-LMCL outperforms existing models and achieves new state-of-the-art performance.
  arXiv  Detail & Related papers  (2023-11-19T16:53:35Z)
• Enhancing and Adversarial: Improve ASR with Speaker Labels [49.73714831258699]
  We propose a novel adaptive gradient reversal layer for stable and effective adversarial training without tuning effort. Detailed analysis and experimental verification are conducted to show the optimal positions in the ASR neural network (NN) to apply speaker enhancing and adversarial training. Our best speaker-based MTL achieves 7% relative improvement on the Switchboard Hub5'00 set.
  arXiv  Detail & Related papers  (2022-11-11T17:40:08Z)
• Contrastive Learning for Improving ASR Robustness in Spoken Language Understanding [28.441725610692714]
  This paper focuses on learning utterance representations that are robust to ASR errors using a contrastive objective. Experiments on three benchmark datasets demonstrate the effectiveness of our proposed approach.
  arXiv  Detail & Related papers  (2022-05-02T07:21:21Z)
• Towards Accurate Knowledge Transfer via Target-awareness Representation Disentanglement [56.40587594647692]
  We propose a novel transfer learning algorithm, introducing the idea of Target-awareness REpresentation Disentanglement (TRED) TRED disentangles the relevant knowledge with respect to the target task from the original source model and used as a regularizer during fine-tuning the target model. Experiments on various real world datasets show that our method stably improves the standard fine-tuning by more than 2% in average.
  arXiv  Detail & Related papers  (2020-10-16T17:45:08Z)
• Joint Contextual Modeling for ASR Correction and Language Understanding [60.230013453699975]
  We propose multi-task neural approaches to perform contextual language correction on ASR outputs jointly with language understanding (LU) We show that the error rates of off the shelf ASR and following LU systems can be reduced significantly by 14% relative with joint models trained using small amounts of in-domain data.
  arXiv  Detail & Related papers  (2020-01-28T22:09:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.