Model-Based Approach for Measuring the Fairness in ASR

• URL: http://arxiv.org/abs/2109.09061v1
• Date: Sun, 19 Sep 2021 05:24:01 GMT
• Title: Model-Based Approach for Measuring the Fairness in ASR
• Authors: Zhe Liu, Irina-Elena Veliche, Fuchun Peng
• Abstract summary: We introduce mixed-effects Poisson regression to better measure and interpret any WER difference among subgroups of interest. We demonstrate the validity of proposed model-based approach on both synthetic and real-world speech data.
• Score: 11.076999352942954
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The issue of fairness arises when the automatic speech recognition (ASR) systems do not perform equally well for all subgroups of the population. In any fairness measurement studies for ASR, the open questions of how to control the nuisance factors, how to handle unobserved heterogeneity across speakers, and how to trace the source of any word error rate (WER) gap among different subgroups are especially important - if not appropriately accounted for, incorrect conclusions will be drawn. In this paper, we introduce mixed-effects Poisson regression to better measure and interpret any WER difference among subgroups of interest. Particularly, the presented method can effectively address the three problems raised above and is very flexible to use in practical disparity analyses. We demonstrate the validity of proposed model-based approach on both synthetic and real-world speech data.

Related papers

• Fair Deepfake Detectors Can Generalize [51.21167546843708]
  We show that controlling for confounders (data distribution and model capacity) enables improved generalization via fairness interventions.<n>Motivated by this insight, we propose Demographic Attribute-insensitive Intervention Detection (DAID), a plug-and-play framework composed of: i) Demographic-aware data rebalancing, which employs inverse-propensity weighting and subgroup-wise feature normalization to neutralize distributional biases; and ii) Demographic-agnostic feature aggregation, which uses a novel alignment loss to suppress sensitive-attribute signals.<n>DAID consistently achieves superior performance in both fairness and generalization compared to several state-of-the-art
  arXiv  Detail & Related papers  (2025-07-03T14:10:02Z)
• Quantifying Fairness in LLMs Beyond Tokens: A Semantic and Statistical Perspective [13.739343897204568]
  Large Language Models (LLMs) often generate responses with inherent biases, undermining their reliability in real-world applications.<n>Existing evaluation methods often overlook biases in long-form responses and the intrinsic variability of LLM outputs.<n>We propose FiSco, a novel statistical framework to evaluate group-level fairness in LLMs by detecting subtle semantic differences in long-form responses across demographic groups.
  arXiv  Detail & Related papers  (2025-06-23T18:31:22Z)
• EMO-Debias: Benchmarking Gender Debiasing Techniques in Multi-Label Speech Emotion Recognition [49.27067541740956]
  EMO-Debias is a large-scale comparison of 13 debiasing methods applied to multi-label SER.<n>Our study encompasses techniques from pre-processing, regularization, adversarial learning, biased learners, and distributionally robust optimization.<n>Our analysis quantifies the trade-offs between fairness and accuracy, identifying which approaches consistently reduce gender performance gaps.
  arXiv  Detail & Related papers  (2025-06-05T05:48:31Z)
• Understanding challenges to the interpretation of disaggregated evaluations of algorithmic fairness [47.31249332115595]
  We show that equal performance across subgroups is an unreliable measure of fairness when data are representative of relevant populations but reflective of real-world disparities.<n>Our framework suggests complementing disaggregated evaluations with explicit causal assumptions and analysis to control for confounding and distribution shift.
  arXiv  Detail & Related papers  (2025-06-04T17:40:31Z)
• Uncovering Fairness through Data Complexity as an Early Indicator [0.7783262415147651]
  There is no study on how disparities in classification complexity between privileged and unprivileged groups could influence the fairness of solutions. In this work, we focus on synthetic datasets designed to capture a variety of biases ranging from historical bias to measurement and representational bias. We then apply association rule mining to identify patterns that link disproportionate complexity differences between groups with fairness-related outcomes.
  arXiv  Detail & Related papers  (2025-04-08T11:28:40Z)
• Unifying Causal Representation Learning with the Invariance Principle [21.375611599649716]
  Causal representation learning aims at recovering latent causal variables from high-dimensional observations. Our main contribution is to show that many existing causal representation learning approaches methodologically align the representation to known data symmetries.
  arXiv  Detail & Related papers  (2024-09-04T14:51:36Z)
• Reconciling Heterogeneous Effects in Causal Inference [44.99833362998488]
  We apply the Reconcile algorithm for model multiplicity in machine learning to reconcile heterogeneous effects in causal inference. Our results have tangible implications for ensuring fair outcomes in high-stakes such as healthcare, insurance, and housing.
  arXiv  Detail & Related papers  (2024-06-05T18:43:46Z)
• DINER: Debiasing Aspect-based Sentiment Analysis with Multi-variable Causal Inference [21.929902181609936]
  We propose a novel framework based on multi-variable causal inference for debiasing ABSA. For the review branch, the bias is modeled as indirect confounding from context, where backdoor adjustment intervention is employed for debiasing. For the aspect branch, the bias is described as a direct correlation with labels, where counterfactual reasoning is adopted for debiasing.
  arXiv  Detail & Related papers  (2024-03-02T10:38:31Z)
• Nonparametric Identifiability of Causal Representations from Unknown Interventions [63.1354734978244]
  We study causal representation learning, the task of inferring latent causal variables and their causal relations from mixtures of the variables. Our goal is to identify both the ground truth latents and their causal graph up to a set of ambiguities which we show to be irresolvable from interventional data.
  arXiv  Detail & Related papers  (2023-06-01T10:51:58Z)
• Fairness Improves Learning from Noisily Labeled Long-Tailed Data [119.0612617460727]
  Long-tailed and noisily labeled data frequently appear in real-world applications and impose significant challenges for learning. We introduce the Fairness Regularizer (FR), inspired by regularizing the performance gap between any two sub-populations. We show that the introduced fairness regularizer improves the performances of sub-populations on the tail and the overall learning performance.
  arXiv  Detail & Related papers  (2023-03-22T03:46:51Z)
• Fairness via Adversarial Attribute Neighbourhood Robust Learning [49.93775302674591]
  We propose a principled underlineRobust underlineAdversarial underlineAttribute underlineNeighbourhood (RAAN) loss to debias the classification head.
  arXiv  Detail & Related papers  (2022-10-12T23:39:28Z)
• Beyond IID: data-driven decision-making in heterogeneous environments [8.714718004930363]
  We study a data-driven decision-making framework in which historical samples are generated from unknown and different distributions. This work aims at analyzing the performance of central data-driven policies but also near-optimal ones.
  arXiv  Detail & Related papers  (2022-06-20T08:43:43Z)
• Addressing Fairness, Bias and Class Imbalance in Machine Learning: the FBI-loss [11.291571222801027]
  We propose a unified loss correction to address issues related to Fairness, Biases and Imbalances (FBI-loss) The correction capabilities of the proposed approach are assessed on three real-world benchmarks.
  arXiv  Detail & Related papers  (2021-05-13T15:01:14Z)
• Can Active Learning Preemptively Mitigate Fairness Issues? [66.84854430781097]
  dataset bias is one of the prevailing causes of unfairness in machine learning. We study whether models trained with uncertainty-based ALs are fairer in their decisions with respect to a protected class. We also explore the interaction of algorithmic fairness methods such as gradient reversal (GRAD) and BALD.
  arXiv  Detail & Related papers  (2021-04-14T14:20:22Z)
• Causal Feature Selection for Algorithmic Fairness [61.767399505764736]
  We consider fairness in the integration component of data management. We propose an approach to identify a sub-collection of features that ensure the fairness of the dataset.
  arXiv  Detail & Related papers  (2020-06-10T20:20:10Z)
• Learning Overlapping Representations for the Estimation of Individualized Treatment Effects [97.42686600929211]
  Estimating the likely outcome of alternatives from observational data is a challenging problem. We show that algorithms that learn domain-invariant representations of inputs are often inappropriate. We develop a deep kernel regression algorithm and posterior regularization framework that substantially outperforms the state-of-the-art on a variety of benchmarks data sets.
  arXiv  Detail & Related papers  (2020-01-14T12:56:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.