Reexamining Racial Disparities in Automatic Speech Recognition Performance: The Role of Confounding by Provenance

• URL: http://arxiv.org/abs/2407.13982v1
• Date: Fri, 19 Jul 2024 02:14:17 GMT
• Title: Reexamining Racial Disparities in Automatic Speech Recognition Performance: The Role of Confounding by Provenance
• Authors: Changye Li, Trevor Cohen, Serguei Pakhomov,
• Abstract summary: It is important that automatic speech recognition (ASR) models and their use is fair and equitable. The current study seeks to understand the factors underlying this disparity by examining the performance of the current state-of-the-art neural network based ASR system.
• Score: 7.882996636086014
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Automatic speech recognition (ASR) models trained on large amounts of audio data are now widely used to convert speech to written text in a variety of applications from video captioning to automated assistants used in healthcare and other domains. As such, it is important that ASR models and their use is fair and equitable. Prior work examining the performance of commercial ASR systems on the Corpus of Regional African American Language (CORAAL) demonstrated significantly worse ASR performance on African American English (AAE). The current study seeks to understand the factors underlying this disparity by examining the performance of the current state-of-the-art neural network based ASR system (Whisper, OpenAI) on the CORAAL dataset. Two key findings have been identified as a result of the current study. The first confirms prior findings of significant dialectal variation even across neighboring communities, and worse ASR performance on AAE that can be improved to some extent with fine-tuning of ASR models. The second is a novel finding not discussed in prior work on CORAAL: differences in audio recording practices within the dataset have a significant impact on ASR accuracy resulting in a ``confounding by provenance'' effect in which both language use and recording quality differ by study location. These findings highlight the need for further systematic investigation to disentangle the effects of recording quality and inherent linguistic diversity when examining the fairness and bias present in neural ASR models, as any bias in ASR accuracy may have negative downstream effects on disparities in various domains of life in which ASR technology is used.

Related papers

• Quantification of stylistic differences in human- and ASR-produced transcripts of African American English [1.8021379035665333]
  Stylistic differences, such as verbatim vs non-verbatim, can play a significant role in ASR performance evaluation. We categorize the kinds of stylistic differences between 6 transcription versions, 4 human- and 2 ASR-produced, of 10 hours of African American English speech. We investigate the interactions of these categories with how well transcripts can be compared via word error rate.
  arXiv  Detail & Related papers  (2024-09-04T20:18:59Z)
• Measuring the Accuracy of Automatic Speech Recognition Solutions [4.99320937849508]
  Automatic Speech Recognition (ASR) is now a part of many popular applications. We measured the performance of eleven common ASR services with recordings of Higher Education lectures. Our results show that accuracy ranges widely between vendors and for the individual audio samples. We also measured a significant lower quality for streaming ASR, which is used for live events.
  arXiv  Detail & Related papers  (2024-08-29T06:38:55Z)
• You don't understand me!: Comparing ASR results for L1 and L2 speakers of Swedish [0.5249805590164903]
  We focus on the gap in performance between recognition results for native and non-native, read and spontaneous, Swedish utterances transcribed by different ASR services. We compare the recognition results using Word Error Rate and analyze the linguistic factors that may generate the observed transcription errors.
  arXiv  Detail & Related papers  (2024-05-22T06:24:55Z)
• AV-RIR: Audio-Visual Room Impulse Response Estimation [49.469389715876915]
  Accurate estimation of Room Impulse Response (RIR) is important for speech processing and AR/VR applications. We propose AV-RIR, a novel multi-modal multi-task learning approach to accurately estimate the RIR from a given reverberant speech signal and visual cues of its corresponding environment.
  arXiv  Detail & Related papers  (2023-11-30T22:58:30Z)
• On the Efficacy and Noise-Robustness of Jointly Learned Speech Emotion and Automatic Speech Recognition [6.006652562747009]
  We investigate a joint ASR-SER learning approach in a low-resource setting. Joint learning can improve ASR word error rate (WER) and SER classification accuracy by 10.7% and 2.3% respectively. Overall, the joint ASR-SER approach yielded more noise-resistant models than the independent ASR and SER approaches.
  arXiv  Detail & Related papers  (2023-05-21T18:52:21Z)
• A Deep Learning System for Domain-specific Speech Recognition [0.0]
  The author works with pre-trained DeepSpeech2 and Wav2Vec2 acoustic models to develop benefit-specific ASR systems. The best performance comes from a fine-tuned Wav2Vec2-Large-LV60 acoustic model with an external KenLM. The viability of using error prone ASR transcriptions as part of spoken language understanding (SLU) is also investigated.
  arXiv  Detail & Related papers  (2023-03-18T22:19:09Z)
• End-to-End Speech Recognition: A Survey [68.35707678386949]
  The goal of this survey is to provide a taxonomy of E2E ASR models and corresponding improvements. All relevant aspects of E2E ASR are covered in this work, accompanied by discussions of performance and deployment opportunities.
  arXiv  Detail & Related papers  (2023-03-03T01:46:41Z)
• Sequence-level self-learning with multiple hypotheses [53.04725240411895]
  We develop new self-learning techniques with an attention-based sequence-to-sequence (seq2seq) model for automatic speech recognition (ASR) In contrast to conventional unsupervised learning approaches, we adopt the emphmulti-task learning (MTL) framework. Our experiment results show that our method can reduce the WER on the British speech data from 14.55% to 10.36% compared to the baseline model trained with the US English data only.
  arXiv  Detail & Related papers  (2021-12-10T20:47:58Z)
• Improving Readability for Automatic Speech Recognition Transcription [50.86019112545596]
  We propose a novel NLP task called ASR post-processing for readability (APR) APR aims to transform the noisy ASR output into a readable text for humans and downstream tasks while maintaining the semantic meaning of the speaker. We compare fine-tuned models based on several open-sourced and adapted pre-trained models with the traditional pipeline method.
  arXiv  Detail & Related papers  (2020-04-09T09:26:42Z)
• Improving noise robust automatic speech recognition with single-channel time-domain enhancement network [100.1041336974175]
  We show that a single-channel time-domain denoising approach can significantly improve ASR performance. We show that single-channel noise reduction can still improve ASR performance.
  arXiv  Detail & Related papers  (2020-03-09T09:36:31Z)
• Joint Contextual Modeling for ASR Correction and Language Understanding [60.230013453699975]
  We propose multi-task neural approaches to perform contextual language correction on ASR outputs jointly with language understanding (LU) We show that the error rates of off the shelf ASR and following LU systems can be reduced significantly by 14% relative with joint models trained using small amounts of in-domain data.
  arXiv  Detail & Related papers  (2020-01-28T22:09:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.