Evaluating Bias in Phoneme-Based Automatic Speech Recognition Systems: An Analysis of IPA Transcription Models

• URL: http://arxiv.org/abs/2606.11639v1
• Date: Wed, 10 Jun 2026 04:00:44 GMT
• Title: Evaluating Bias in Phoneme-Based Automatic Speech Recognition Systems: An Analysis of IPA Transcription Models
• Abstract summary: We evaluate the performance of two state-of-the-art open-source ASR systems, WhisperIPA and ZIPA, that generate International Phonetic Alphabet (IPA) transcriptions.<n>Our analysis examines how performance varies across languages and demographic groups such as gender, accent, ethnicity, and age.
• Score: 2.857724571923244
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The popularization of automatic speech recognition (ASR) systems has increased exploration of the demographic biases related to race, age, gender, and accent, often formed from imbalanced training data. Most of these studies focused on standard grapheme-based ASR systems with comparatively little emphasis on phoneme-based systems, such as models that produce International Phonetic Alphabet (IPA) representations. As ASR systems shift toward multilingual support and low-resource language modeling, IPA-based layers serve as a critical, language-agnostic foundation. In this study, we evaluate the performance of two state-of-the-art open-source ASR systems, WhisperIPA and ZIPA, that generate IPA transcriptions across diverse accents and language sources. Our evaluation includes existing multilingual speech corpora and demographically annotated English-language corpora. We measure model performance by comparing model-generated IPA transcriptions against grapheme-to-phoneme (G2P) systems using both standard phoneme error rate (PER) and a proposed Soft PER metric that tolerates linguistically similar phoneme substitutions. Our analysis examines how performance varies across languages and demographic groups such as gender, accent, ethnicity, and age, revealing persistent disparities even after accounting for acceptable phonemic variation. These findings provide insight into potential sources of bias and inform the development of more inclusive and linguistically robust phoneme-based ASR systems. Our code and data will be made publicly available to the community.

Related papers

• Linguistic Distance Segregates Latent Representations in Automatic Speech Recognition Systems [3.1670140283390276]
  This paper investigates the relationship between first language background and English ASR performance.<n>Through empirical analysis, we observe that the correlation between speakers' L1 distance and ASR error rates yields a systematic effect on English Speech.
  arXiv  Detail & Related papers  (2026-08-31T14:16:21Z)
• PRiSM: Benchmarking Phone Realization in Speech Models [70.82595415252682]
  Phone recognition (PR) serves as the atomic interface for language-agnostic modeling for cross-lingual speech processing and phonetic analysis.<n>We introduce PRiSM, the first open-source benchmark designed to expose blind spots in phonetic perception.
  arXiv  Detail & Related papers  (2026-01-20T15:00:36Z)
• A Sociophonetic Analysis of Racial Bias in Commercial ASR Systems Using the Pacific Northwest English Corpus [6.361208877327219]
  This paper presents a systematic evaluation of racial bias in four major commercial automatic speech recognition (ASR) systems.<n>We analyze transcription accuracy across speakers from four ethnic backgrounds (African American, Caucasian American, ChicanX, and Yakama) and examine how sociophonetic variation contributes to differential system performance.
  arXiv  Detail & Related papers  (2025-10-26T02:19:40Z)
• Accent-Invariant Automatic Speech Recognition via Saliency-Driven Spectrogram Masking [1.108292291257035]
  We propose an accent-invariant ASR framework that integrates accent and dialect classification into the recognition pipeline.<n>Our approach involves training a spectrogram-based classifier to capture accent-specific cues, masking the regions most influential to its predictions, and using the masked spectrograms for data augmentation.<n>For Persian, we introduce a newly collected dataset spanning multiple regional accents, establishing the first systematic benchmark for accent variation in Persian ASR.
  arXiv  Detail & Related papers  (2025-10-10T16:41:53Z)
• An Initial Investigation of Language Adaptation for TTS Systems under Low-resource Scenarios [76.11409260727459]
  This paper explores the language adaptation capability of ZMM-TTS, a recent SSL-based multilingual TTS system. We demonstrate that the similarity in phonetics between the pre-training and target languages, as well as the language category, affects the target language's adaptation performance.
  arXiv  Detail & Related papers  (2024-06-13T08:16:52Z)
• The Balancing Act: Unmasking and Alleviating ASR Biases in Portuguese [5.308321515594125]
  This study is dedicated to a comprehensive exploration of the Whisper and MMS systems. Our investigation encompasses various categories, including gender, age, skin tone color, and geo-location. We empirically show that oversampling techniques alleviate such stereotypical biases.
  arXiv  Detail & Related papers  (2024-02-12T09:35:13Z)
• Language-universal phonetic encoder for low-resource speech recognition [28.21805271848413]
  We leverage International Phonetic Alphabet (IPA) based language-universal phonetic model to improve low-resource ASR performances. Our approach and adaptation are effective on extremely low-resource languages, even within domain- and language-mismatched scenarios.
  arXiv  Detail & Related papers  (2023-05-19T10:24:30Z)
• Investigating the Sensitivity of Automatic Speech Recognition Systems to Phonetic Variation in L2 Englishes [3.198144010381572]
  This work demonstrates a method of probing an ASR system to discover how it handles phonetic variation across a number of L2 Englishes. It is demonstrated that the behaviour of the ASR is systematic and consistent across speakers with similar spoken varieties.
  arXiv  Detail & Related papers  (2023-05-12T11:29:13Z)
• Discovering Phonetic Inventories with Crosslingual Automatic Speech Recognition [71.49308685090324]
  This paper investigates the influence of different factors (i.e., model architecture, phonotactic model, type of speech representation) on phone recognition in an unknown language. We find that unique sounds, similar sounds, and tone languages remain a major challenge for phonetic inventory discovery.
  arXiv  Detail & Related papers  (2022-01-26T22:12:55Z)
• Acoustics Based Intent Recognition Using Discovered Phonetic Units for Low Resource Languages [51.0542215642794]
  We propose a novel acoustics based intent recognition system that uses discovered phonetic units for intent classification. We present results for two languages families - Indic languages and Romance languages, for two different intent recognition tasks.
  arXiv  Detail & Related papers  (2020-11-07T00:35:31Z)
• That Sounds Familiar: an Analysis of Phonetic Representations Transfer Across Languages [72.9927937955371]
  We use the resources existing in other languages to train a multilingual automatic speech recognition model. We observe significant improvements across all languages in the multilingual setting, and stark degradation in the crosslingual setting. Our analysis uncovered that even the phones that are unique to a single language can benefit greatly from adding training data from other languages.
  arXiv  Detail & Related papers  (2020-05-16T22:28:09Z)
• Universal Phone Recognition with a Multilingual Allophone System [135.2254086165086]
  We propose a joint model of language-independent phone and language-dependent phoneme distributions. In multilingual ASR experiments over 11 languages, we find that this model improves testing performance by 2% phoneme error rate absolute. Our recognizer achieves phone accuracy improvements of more than 17%, moving a step closer to speech recognition for all languages in the world.
  arXiv  Detail & Related papers  (2020-02-26T21:28:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.