Facets of Disparate Impact: Evaluating Legally Consistent Bias in Machine Learning

• URL: http://arxiv.org/abs/2505.05471v1
• Date: Thu, 08 May 2025 17:58:49 GMT
• Title: Facets of Disparate Impact: Evaluating Legally Consistent Bias in Machine Learning
• Authors: Jarren Briscoe, Assefaw Gebremedhin,
• Abstract summary: Using current legal standards, we define bias through the lens of marginal benefits and objective testing.<n>This index combines the contextual nuances of objective testing with metric stability, providing a legally consistent and reliable measure.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Leveraging current legal standards, we define bias through the lens of marginal benefits and objective testing with the novel metric "Objective Fairness Index". This index combines the contextual nuances of objective testing with metric stability, providing a legally consistent and reliable measure. Utilizing the Objective Fairness Index, we provide fresh insights into sensitive machine learning applications, such as COMPAS (recidivism prediction), highlighting the metric's practical and theoretical significance. The Objective Fairness Index allows one to differentiate between discriminatory tests and systemic disparities.

Related papers

• Revisiting LLM Evaluation through Mechanism Interpretability: a New Metric and Model Utility Law [99.56567010306807]
  Large Language Models (LLMs) have become indispensable across academia, industry, and daily applications.<n>We propose a novel metric, the Model Utilization Index (MUI), which introduces mechanism interpretability techniques to complement traditional performance metrics.
  arXiv  Detail & Related papers  (2025-04-10T04:09:47Z)
• Fairness Evaluation with Item Response Theory [10.871079276188649]
  This paper proposes a novel Fair-IRT framework to evaluate fairness in Machine Learning (ML) models. Detailed explanations for item characteristic curves (ICCs) are provided for particular individuals. Experiments demonstrate the effectiveness of this framework as a fairness evaluation tool.
  arXiv  Detail & Related papers  (2024-10-20T22:25:20Z)
• The Fragility of Fairness: Causal Sensitivity Analysis for Fair Machine Learning [34.50562695587344]
  We adapt tools from causal sensitivity analysis to the FairML context. We analyze the sensitivity of the most common parity metrics under 3 varieties of classifier. We show that causal sensitivity analysis provides a powerful and necessary toolkit for gauging the informativeness of parity metric evaluations.
  arXiv  Detail & Related papers  (2024-10-12T17:28:49Z)
• Goodhart's Law Applies to NLP's Explanation Benchmarks [57.26445915212884]
  We critically examine two sets of metrics: the ERASER metrics (comprehensiveness and sufficiency) and the EVAL-X metrics. We show that we can inflate a model's comprehensiveness and sufficiency scores dramatically without altering its predictions or explanations on in-distribution test inputs. Our results raise doubts about the ability of current metrics to guide explainability research, underscoring the need for a broader reassessment of what precisely these metrics are intended to capture.
  arXiv  Detail & Related papers  (2023-08-28T03:03:03Z)
• Towards Better Fairness-Utility Trade-off: A Comprehensive Measurement-Based Reinforcement Learning Framework [7.8940121707748245]
  How to ensure machine learning's fairness while maintaining its utility is a challenging but crucial issue. We propose CFU (Comprehensive Fairness-Utility), a reinforcement learning-based framework, to efficiently improve the fairness-utility trade-off. CFU outperforms all state-of-the-art techniques and has witnessed a 37.5% improvement on average.
  arXiv  Detail & Related papers  (2023-07-21T06:34:41Z)
• Practical Approaches for Fair Learning with Multitype and Multivariate Sensitive Attributes [70.6326967720747]
  It is important to guarantee that machine learning algorithms deployed in the real world do not result in unfairness or unintended social consequences. We introduce FairCOCCO, a fairness measure built on cross-covariance operators on reproducing kernel Hilbert Spaces. We empirically demonstrate consistent improvements against state-of-the-art techniques in balancing predictive power and fairness on real-world datasets.
  arXiv  Detail & Related papers  (2022-11-11T11:28:46Z)
• Fair Machine Learning in Healthcare: A Review [90.22219142430146]
  We analyze the intersection of fairness in machine learning and healthcare disparities. We provide a critical review of the associated fairness metrics from a machine learning standpoint. We propose several new research directions that hold promise for developing ethical and equitable ML applications in healthcare.
  arXiv  Detail & Related papers  (2022-06-29T04:32:10Z)
• Conditional Supervised Contrastive Learning for Fair Text Classification [59.813422435604025]
  We study learning fair representations that satisfy a notion of fairness known as equalized odds for text classification via contrastive learning. Specifically, we first theoretically analyze the connections between learning representations with a fairness constraint and conditional supervised contrastive objectives.
  arXiv  Detail & Related papers  (2022-05-23T17:38:30Z)
• Measuring Fairness of Text Classifiers via Prediction Sensitivity [63.56554964580627]
  ACCUMULATED PREDICTION SENSITIVITY measures fairness in machine learning models based on the model's prediction sensitivity to perturbations in input features. We show that the metric can be theoretically linked with a specific notion of group fairness (statistical parity) and individual fairness.
  arXiv  Detail & Related papers  (2022-03-16T15:00:33Z)
• Fair Tree Learning [0.15229257192293202]
  Various optimisation criteria combine classification performance with a fairness metric. Current fair decision tree methods only optimise for a fixed threshold on both the classification task as well as the fairness metric. We propose a threshold-independent fairness metric termed uniform demographic parity, and a derived splitting criterion entitled SCAFF -- Splitting Criterion AUC for Fairness.
  arXiv  Detail & Related papers  (2021-10-18T13:40:25Z)
• ReMP: Rectified Metric Propagation for Few-Shot Learning [67.96021109377809]
  A rectified metric space is learned to maintain the metric consistency from training to testing. Numerous analyses indicate that a simple modification of the objective can yield substantial performance gains. The proposed ReMP is effective and efficient, and outperforms the state of the arts on various standard few-shot learning datasets.
  arXiv  Detail & Related papers  (2020-12-02T00:07:53Z)
• GO FIGURE: A Meta Evaluation of Factuality in Summarization [131.1087461486504]
  We introduce GO FIGURE, a meta-evaluation framework for evaluating factuality evaluation metrics. Our benchmark analysis on ten factuality metrics reveals that our framework provides a robust and efficient evaluation. It also reveals that while QA metrics generally improve over standard metrics that measure factuality across domains, performance is highly dependent on the way in which questions are generated.
  arXiv  Detail & Related papers  (2020-10-24T08:30:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.