Responsible Benchmarking of Fairness for Automatic Speech Recognition

• URL: http://arxiv.org/abs/2605.10615v1
• Date: Mon, 11 May 2026 14:11:50 GMT
• Title: Responsible Benchmarking of Fairness for Automatic Speech Recognition
• Abstract summary: Many studies have shown automatic speech processing (ASR) systems have unequal performance across speakergroups (SG's)<n>We lay out best practices for benchmarking ASR fairness based on literaturefrom machine learning fairness, social sciences, and speech science.<n>We advocate for as fine-grained an analysis as possibleof the intersectionality of as many demographic variables as are available in the metadata of fairness corpora.
• Score: 44.6396086462419
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Many studies have shown automatic speech processing (ASR) systems have unequal performance across speakergroups (SG's). However, the manner in which such studies arrive at this conclusion is inconsistent. To pave the wayfor more reliable results in future studies, we lay out best practices for benchmarking ASR fairness based on literaturefrom machine learning fairness, social sciences, and speech science. We first describe the importance of preciselythe fairness hypothesis being interrogated, and tailoring fairness metrics to apply specifically to said hypothesis.We then examine several benchmarks used to rate ASR systems on fairness and discuss how their results can bemisconstrued without assiduous oversight into the intersections between SG's. We find that evaluating fairnessbased on single heterogeneous SG's, such as they are defined in fairness benchmarks, can lead to misidentifyingwhich SG's are actually being mistreated by ASR systems. We advocate for as fine-grained an analysis as possibleof the intersectionality of as many demographic variables as are available in the metadata of fairness corpora in orderto tease out such spurious correlations

Related papers

• Does RAG Introduce Unfairness in LLMs? Evaluating Fairness in Retrieval-Augmented Generation Systems [18.926129063000264]
  We aim to empirically evaluate fairness in several Retrieval-Augmented Generation (RAG) methods.<n>Our experimental results indicate that, despite recent advances in utility-driven optimization, fairness issues persist in both the retrieval and generation stages.<n>These findings underscore the need for targeted interventions to address fairness concerns throughout the RAG pipeline.
  arXiv  Detail & Related papers  (2024-09-29T22:04:26Z)
• Towards Fair RAG: On the Impact of Fair Ranking in Retrieval-Augmented Generation [53.285436927963865]
  This paper is the first systematic evaluation of RAG systems that integrate fairness-aware rankings.<n>We show that incorporating fairness-aware retrieval often maintains or even enhances both ranking quality and generation quality.
  arXiv  Detail & Related papers  (2024-09-17T23:10:04Z)
• Evaluating Human Alignment and Model Faithfulness of LLM Rationale [66.75309523854476]
  We study how well large language models (LLMs) explain their generations through rationales. We show that prompting-based methods are less "faithful" than attribution-based explanations.
  arXiv  Detail & Related papers  (2024-06-28T20:06:30Z)
• FairLENS: Assessing Fairness in Law Enforcement Speech Recognition [37.75768315119143]
  We propose a novel and adaptable evaluation method to examine the fairness disparity between different models. We conducted fairness assessments on 1 open-source and 11 commercially available state-of-the-art ASR models.
  arXiv  Detail & Related papers  (2024-05-21T19:23:40Z)
• RankCSE: Unsupervised Sentence Representations Learning via Learning to Rank [54.854714257687334]
  We propose a novel approach, RankCSE, for unsupervised sentence representation learning. It incorporates ranking consistency and ranking distillation with contrastive learning into a unified framework. An extensive set of experiments are conducted on both semantic textual similarity (STS) and transfer (TR) tasks.
  arXiv  Detail & Related papers  (2023-05-26T08:27:07Z)
• Perspectives on Large Language Models for Relevance Judgment [56.935731584323996]
  Large language models (LLMs) claim that they can assist with relevance judgments. It is not clear whether automated judgments can reliably be used in evaluations of retrieval systems.
  arXiv  Detail & Related papers  (2023-04-13T13:08:38Z)
• Fairness meets Cross-Domain Learning: a new perspective on Models and Metrics [80.07271410743806]
  We study the relationship between cross-domain learning (CD) and model fairness. We introduce a benchmark on face and medical images spanning several demographic groups as well as classification and localization tasks. Our study covers 14 CD approaches alongside three state-of-the-art fairness algorithms and shows how the former can outperform the latter.
  arXiv  Detail & Related papers  (2023-03-25T09:34:05Z)
• A Study on Bias and Fairness In Deep Speaker Recognition [25.71206255965502]
  We study the notion of fairness in recent speaker recognition (SR) systems based on 3 popular definitions, namely Statistical Parity, Equalized Odds, and Equal Opportunity. We examine 5 popular neural architectures and 5 commonly used loss functions in training SR systems, while evaluating their fairness against gender and nationality groups.
  arXiv  Detail & Related papers  (2023-03-14T16:08:19Z)
• Error Parity Fairness: Testing for Group Fairness in Regression Tasks [5.076419064097733]
  This work presents error parity as a regression fairness notion and introduces a testing methodology to assess group fairness. It is followed by a suitable permutation test to compare groups on several statistics to explore disparities and identify impacted groups. Overall, the proposed regression fairness testing methodology fills a gap in the fair machine learning literature and may serve as a part of larger accountability assessments and algorithm audits.
  arXiv  Detail & Related papers  (2022-08-16T17:47:20Z)
• Measuring Fairness of Text Classifiers via Prediction Sensitivity [63.56554964580627]
  ACCUMULATED PREDICTION SENSITIVITY measures fairness in machine learning models based on the model's prediction sensitivity to perturbations in input features. We show that the metric can be theoretically linked with a specific notion of group fairness (statistical parity) and individual fairness.
  arXiv  Detail & Related papers  (2022-03-16T15:00:33Z)
• Model-Based Approach for Measuring the Fairness in ASR [11.076999352942954]
  We introduce mixed-effects Poisson regression to better measure and interpret any WER difference among subgroups of interest. We demonstrate the validity of proposed model-based approach on both synthetic and real-world speech data.
  arXiv  Detail & Related papers  (2021-09-19T05:24:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.