Learning Fair Scoring Functions: Bipartite Ranking under ROC-based Fairness Constraints

• URL: http://arxiv.org/abs/2002.08159v4
• Date: Thu, 25 Feb 2021 18:54:20 GMT
• Title: Learning Fair Scoring Functions: Bipartite Ranking under ROC-based Fairness Constraints
• Abstract summary: We study fairness for the problem of learning scoring functions from binary labeled data. We introduce general families of fairness definitions based on the AUC and on ROC curves. We establish bounds generalization for scoring functions learned under such constraints.
• Score: 2.0263791972068628
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Many applications of AI involve scoring individuals using a learned function of their attributes. These predictive risk scores are then used to take decisions based on whether the score exceeds a certain threshold, which may vary depending on the context. The level of delegation granted to such systems in critical applications like credit lending and medical diagnosis will heavily depend on how questions of fairness can be answered. In this paper, we study fairness for the problem of learning scoring functions from binary labeled data, a classic learning task known as bipartite ranking. We argue that the functional nature of the ROC curve, the gold standard measure of ranking accuracy in this context, leads to several ways of formulating fairness constraints. We introduce general families of fairness definitions based on the AUC and on ROC curves, and show that our ROC-based constraints can be instantiated such that classifiers obtained by thresholding the scoring function satisfy classification fairness for a desired range of thresholds. We establish generalization bounds for scoring functions learned under such constraints, design practical learning algorithms and show the relevance our approach with numerical experiments on real and synthetic data.

Related papers

• CLARK: Closed-loop Learning for Adaptive Reasoning over Knowledge Graphs [2.2889981393105554]
  CLARK (Closed-loop Learning for Adaptive Reasoning over Knowledge Graphs) is a framework that integrates knowledge graphs, symbolic rule mining, and probabilistic reasoning.<n>We evaluate CLARK on two medical datasets, analysing both rule quality and downstream classification performance.
  arXiv  Detail & Related papers  (2026-07-22T10:30:45Z)
• fairmetrics: An R package for group fairness evaluation [0.40964539027092906]
  fairmetrics R package offers a user-friendly framework for rigorously evaluating group-based fairness criteria.<n>Group-based fairness criteria assess whether a model is equally accurate or well-calibrated across a set of predefined groups.<n>fairmetrics provides both point and interval estimates for multiple metrics through a convenient wrapper function.
  arXiv  Detail & Related papers  (2025-06-06T17:07:41Z)
• Quantifying Query Fairness Under Unawareness [82.33181164973365]
  We introduce a robust fairness estimator based on quantification that effectively handles multiple sensitive attributes beyond binary classifications.<n>Our method outperforms existing baselines across various sensitive attributes and is the first to establish a reliable protocol for measuring fairness under unawareness.
  arXiv  Detail & Related papers  (2025-06-04T16:31:44Z)
• Enforcing Fairness Where It Matters: An Approach Based on Difference-of-Convex Constraints [12.054667230143803]
  We focus on achieving full fairness across all score ranges by predictive models, ensuring in both high and low-scoring populations.<n>We propose a novel score of interest as the middle where decisions are most contested, while maintaining flexibility in other regions.<n>We introduce two statistical metrics to rigorously evaluate fairness within a given score range.
  arXiv  Detail & Related papers  (2025-05-18T19:50:01Z)
• Fairness-aware Bayes optimal functional classification [7.252020628689346]
  Algorithmic fairness has become a central topic in machine learning.<n>We study the classification of functional data under fairness constraints.<n>We propose a unified framework for fairness-aware functional classification.
  arXiv  Detail & Related papers  (2025-05-14T15:22:09Z)
• Evaluating Human Alignment and Model Faithfulness of LLM Rationale [66.75309523854476]
  We study how well large language models (LLMs) explain their generations through rationales. We show that prompting-based methods are less "faithful" than attribution-based explanations.
  arXiv  Detail & Related papers  (2024-06-28T20:06:30Z)
• Evaluating Generative Language Models in Information Extraction as Subjective Question Correction [49.729908337372436]
  We propose a new evaluation method, SQC-Score. Inspired by the principles in subjective question correction, we propose a new evaluation method, SQC-Score. Results on three information extraction tasks show that SQC-Score is more preferred by human annotators than the baseline metrics.
  arXiv  Detail & Related papers  (2024-04-04T15:36:53Z)
• Learning Fair Classifiers via Min-Max F-divergence Regularization [13.81078324883519]
  We introduce a novel min-max F-divergence regularization framework for learning fair classification models. We show that F-divergence measures possess convexity and differentiability properties. We show that the proposed framework achieves state-of-the-art performance with respect to the trade-off between accuracy and fairness.
  arXiv  Detail & Related papers  (2023-06-28T20:42:04Z)
• Fairness meets Cross-Domain Learning: a new perspective on Models and Metrics [80.07271410743806]
  We study the relationship between cross-domain learning (CD) and model fairness. We introduce a benchmark on face and medical images spanning several demographic groups as well as classification and localization tasks. Our study covers 14 CD approaches alongside three state-of-the-art fairness algorithms and shows how the former can outperform the latter.
  arXiv  Detail & Related papers  (2023-03-25T09:34:05Z)
• I Know Therefore I Score: Label-Free Crafting of Scoring Functions using Constraints Based on Domain Expertise [6.26476800426345]
  We introduce a label-free practical approach to learn a scoring function from multi-dimensional numerical data. The approach incorporates insights and business rules from domain experts in the form of easily observable and specifiable constraints. We convert such constraints into loss functions that are optimized simultaneously while learning the scoring function.
  arXiv  Detail & Related papers  (2022-03-18T17:51:20Z)
• Fair Tree Learning [0.15229257192293202]
  Various optimisation criteria combine classification performance with a fairness metric. Current fair decision tree methods only optimise for a fixed threshold on both the classification task as well as the fairness metric. We propose a threshold-independent fairness metric termed uniform demographic parity, and a derived splitting criterion entitled SCAFF -- Splitting Criterion AUC for Fairness.
  arXiv  Detail & Related papers  (2021-10-18T13:40:25Z)
• Concentration Inequalities for Two-Sample Rank Processes with Application to Bipartite Ranking [0.0]
  The ROC curve is the gold standard for measuring the performance of a test/scoring statistic. Most practical performance measures used in scoring/ranking applications can be viewed as summaries of the ROC curve.
  arXiv  Detail & Related papers  (2021-04-07T06:31:06Z)
• Leveraging Expert Consistency to Improve Algorithmic Decision Support [62.61153549123407]
  We explore the use of historical expert decisions as a rich source of information that can be combined with observed outcomes to narrow the construct gap. We propose an influence function-based methodology to estimate expert consistency indirectly when each case in the data is assessed by a single expert. Our empirical evaluation, using simulations in a clinical setting and real-world data from the child welfare domain, indicates that the proposed approach successfully narrows the construct gap.
  arXiv  Detail & Related papers  (2021-01-24T05:40:29Z)
• Towards Model-Agnostic Post-Hoc Adjustment for Balancing Ranking Fairness and Algorithm Utility [54.179859639868646]
  Bipartite ranking aims to learn a scoring function that ranks positive individuals higher than negative ones from labeled data. There have been rising concerns on whether the learned scoring function can cause systematic disparity across different protected groups. We propose a model post-processing framework for balancing them in the bipartite ranking scenario.
  arXiv  Detail & Related papers  (2020-06-15T10:08:39Z)
• Uncertainty-aware Score Distribution Learning for Action Quality Assessment [91.05846506274881]
  We propose an uncertainty-aware score distribution learning (USDL) approach for action quality assessment (AQA) Specifically, we regard an action as an instance associated with a score distribution, which describes the probability of different evaluated scores. Under the circumstance where fine-grained score labels are available, we devise a multi-path uncertainty-aware score distributions learning (MUSDL) method to explore the disentangled components of a score.
  arXiv  Detail & Related papers  (2020-06-13T15:41:29Z)
• Fairness by Explicability and Adversarial SHAP Learning [0.0]
  We propose a new definition of fairness that emphasises the role of an external auditor and model explicability. We develop a framework for mitigating model bias using regularizations constructed from the SHAP values of an adversarial surrogate model. We demonstrate our approaches using gradient and adaptive boosting on: a synthetic dataset, the UCI Adult (Census) dataset and a real-world credit scoring dataset.
  arXiv  Detail & Related papers  (2020-03-11T14:36:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.