Towards Fair ASR For Second Language Speakers Using Fairness Prompted Finetuning

• URL: http://arxiv.org/abs/2510.18374v1
• Date: Tue, 21 Oct 2025 07:45:21 GMT
• Title: Towards Fair ASR For Second Language Speakers Using Fairness Prompted Finetuning
• Abstract summary: We address the challenge of building fair English ASR systems for second-language speakers.<n>Our analysis of widely used ASR models, Whisper and Seamless-M4T, reveals large fluctuations in word error rate (WER) across 26 accent groups.<n>To mitigate this, we propose fairness-prompted finetuning with lightweight adapters.
• Score: 37.91306016310542
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this work, we address the challenge of building fair English ASR systems for second-language speakers. Our analysis of widely used ASR models, Whisper and Seamless-M4T, reveals large fluctuations in word error rate (WER) across 26 accent groups, indicating significant fairness gaps. To mitigate this, we propose fairness-prompted finetuning with lightweight adapters, incorporating Spectral Decoupling (SD), Group Distributionally Robust Optimization (Group-DRO), and Invariant Risk Minimization (IRM). Our proposed fusion of traditional empirical risk minimization (ERM) with cross-entropy and fairness-driven objectives (SD, Group DRO, and IRM) enhances fairness across accent groups while maintaining overall recognition accuracy. In terms of macro-averaged word error rate, our approach achieves a relative improvement of 58.7% and 58.5% over the large pretrained Whisper and SeamlessM4T, and 9.7% and 7.8% over them, finetuning with standard empirical risk minimization with cross-entropy loss.

Related papers

• Mega-ASR: Towards In-the-wild^2 Speech Recognition via Scaling up Real-world Acoustic Simulation [78.39988331831077]
  Mega-ASR combines scalable compound-data construction with progressive acoustic-to-semantic optimization.<n>We show Mega-ASR achieves significant advantages over prior state-of-the-art systems on adverse-condition ASR benchmarks.
  arXiv  Detail & Related papers  (2026-05-19T13:26:51Z)
• STaR-DRO: Stateful Tsallis Reweighting for Group-Robust Structured Prediction [0.20165644958637055]
  We present a two-part framework for controllable inference and robust fine-tuning.<n>First, we introduce a task-agnostic prompting strategy to address format drift, label ambiguity, evidence hallucination, and metadata-conditioned confusion.<n>Second, we introduce STaR-DRO, a stateful robust optimization method for group heterogeneity.
  arXiv  Detail & Related papers  (2026-04-09T21:58:42Z)
• Distributionally Robust Token Optimization in RLHF [5.00213782531351]
  Large Language Models (LLMs) tend to respond correctly to prompts that align to the data they were trained and fine-tuned on.<n>Small shifts in wording, format, or language can trigger surprisingly large failures, especially on multi-step reasoning problems.<n>We propose a Distributionally Robust Token Optimization approach, which combines token-level Reinforcement Learning from Human Feedback with Distributionally Robust Optimization.
  arXiv  Detail & Related papers  (2026-03-27T21:36:32Z)
• Proficiency-Aware Adaptation and Data Augmentation for Robust L2 ASR [1.9051696778411429]
  General-purpose ASR underperforms for atypical speakers, such as L2 learners, reinforcing bias and limiting use in education and accessibility.<n>We show that naive fine-tuning of Whisper reduces average WER but simultaneously widens disparities and disproportionately harms lower-level learners.<n>We propose proficiency-aware multitask learning, jointly optimizing ASR with proficiency classification, and (ii) targeted augmentation, applying spectrogram masking to low-proficiency speech to counter imbalance.
  arXiv  Detail & Related papers  (2025-10-12T18:20:58Z)
• Mind the Confidence Gap: Overconfidence, Calibration, and Distractor Effects in Large Language Models [0.6091702876917281]
  Large Language Models (LLMs) show remarkable proficiency in natural language tasks.<n>Overconfidence-misalignment between predicted confidence and true correctness poses significant risks in critical decision-making applications.<n>We present a comprehensive analysis on calibration in LLMs across nine LLMs and three factual Question-Answering datasets.
  arXiv  Detail & Related papers  (2025-02-16T07:46:09Z)
• CTC-DRO: Robust Optimization for Reducing Language Disparities in Speech Recognition [61.925178250628825]
  Group distributionally robust optimization (group DRO) addresses this problem by minimizing the worst-group loss, but it fails when group losses misrepresent performance differences between groups.<n>We present CTC-DRO, which addresses the shortcomings of the group DRO objective by smoothing the group weight update to prevent overemphasis on consistently high-loss groups.
  arXiv  Detail & Related papers  (2025-02-03T19:29:42Z)
• Modeling the Q-Diversity in a Min-max Play Game for Robust Optimization [61.39201891894024]
  Group distributionally robust optimization (group DRO) can minimize the worst-case loss over pre-defined groups. We reformulate the group DRO framework by proposing Q-Diversity. Characterized by an interactive training mode, Q-Diversity relaxes the group identification from annotation into direct parameterization.
  arXiv  Detail & Related papers  (2023-05-20T07:02:27Z)
• On-the-Fly Feature Based Rapid Speaker Adaptation for Dysarthric and Elderly Speech Recognition [53.17176024917725]
  Scarcity of speaker-level data limits the practical use of data-intensive model based speaker adaptation methods. This paper proposes two novel forms of data-efficient, feature-based on-the-fly speaker adaptation methods.
  arXiv  Detail & Related papers  (2022-03-28T09:12:24Z)
• Distributionally Robust Multiclass Classification and Applications in Deep Image Classifiers [3.179831861897336]
  We develop a Distributionally Robust Optimization (DRO) formulation for Multiclass Logistic Regression (MLR) We demonstrate reductions in test error rate by up to 83.5% and loss by up to 91.3% compared with baseline methods, by adopting a novel random training method.
  arXiv  Detail & Related papers  (2021-09-27T02:58:19Z)
• Just Train Twice: Improving Group Robustness without Training Group Information [101.84574184298006]
  Standard training via empirical risk minimization can produce models that achieve high accuracy on average but low accuracy on certain groups. Prior approaches that achieve high worst-group accuracy, like group distributionally robust optimization (group DRO) require expensive group annotations for each training point. We propose a simple two-stage approach, JTT, that first trains a standard ERM model for several epochs, and then trains a second model that upweights the training examples that the first model misclassified.
  arXiv  Detail & Related papers  (2021-07-19T17:52:32Z)
• Joint Contextual Modeling for ASR Correction and Language Understanding [60.230013453699975]
  We propose multi-task neural approaches to perform contextual language correction on ASR outputs jointly with language understanding (LU) We show that the error rates of off the shelf ASR and following LU systems can be reduced significantly by 14% relative with joint models trained using small amounts of in-domain data.
  arXiv  Detail & Related papers  (2020-01-28T22:09:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.