Standardized Interpretable Fairness Measures for Continuous Risk Scores

• URL: http://arxiv.org/abs/2308.11375v2
• Date: Thu, 29 Aug 2024 14:45:26 GMT
• Title: Standardized Interpretable Fairness Measures for Continuous Risk Scores
• Authors: Ann-Kristin Becker, Oana Dumitrasc, Klaus Broelemann,
• Abstract summary: We propose a standardized version of fairness measures for continuous scores with a reasonable interpretation based on the Wasserstein distance. Our measures are easily computable and well suited for quantifying and interpreting the strength of group disparities as well as for comparing biases across different models, datasets, or time points.
• Score: 4.192037827105842
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We propose a standardized version of fairness measures for continuous scores with a reasonable interpretation based on the Wasserstein distance. Our measures are easily computable and well suited for quantifying and interpreting the strength of group disparities as well as for comparing biases across different models, datasets, or time points. We derive a link between the different families of existing fairness measures for scores and show that the proposed standardized fairness measures outperform ROC-based fairness measures because they are more explicit and can quantify significant biases that ROC-based fairness measures miss.

Related papers

• Goodhart's Law Applies to NLP's Explanation Benchmarks [57.26445915212884]
  We critically examine two sets of metrics: the ERASER metrics (comprehensiveness and sufficiency) and the EVAL-X metrics. We show that we can inflate a model's comprehensiveness and sufficiency scores dramatically without altering its predictions or explanations on in-distribution test inputs. Our results raise doubts about the ability of current metrics to guide explainability research, underscoring the need for a broader reassessment of what precisely these metrics are intended to capture.
  arXiv  Detail & Related papers  (2023-08-28T03:03:03Z)
• On (assessing) the fairness of risk score models [2.0646127669654826]
  Risk models are of interest for a number of reasons, including the fact that they communicate uncertainty about the potential outcomes to users. We identify the provision of similar value to different groups as a key desideratum for risk score fairness. We introduce a novel calibration error metric that is less sample size-biased than previously proposed metrics.
  arXiv  Detail & Related papers  (2023-02-17T12:45:51Z)
• Learning Informative Representation for Fairness-aware Multivariate Time-series Forecasting: A Group-based Perspective [50.093280002375984]
  Performance unfairness among variables widely exists in multivariate time series (MTS) forecasting models. We propose a novel framework, named FairFor, for fairness-aware MTS forecasting.
  arXiv  Detail & Related papers  (2023-01-27T04:54:12Z)
• Measuring Fairness of Text Classifiers via Prediction Sensitivity [63.56554964580627]
  ACCUMULATED PREDICTION SENSITIVITY measures fairness in machine learning models based on the model's prediction sensitivity to perturbations in input features. We show that the metric can be theoretically linked with a specific notion of group fairness (statistical parity) and individual fairness.
  arXiv  Detail & Related papers  (2022-03-16T15:00:33Z)
• Gradual (In)Compatibility of Fairness Criteria [0.0]
  Impossibility results show that important fairness measures cannot be satisfied at the same time under reasonable assumptions. This paper explores whether we can satisfy and/or improve these fairness measures simultaneously to a certain degree.
  arXiv  Detail & Related papers  (2021-09-09T16:37:30Z)
• Fair Mixup: Fairness via Interpolation [28.508444261249423]
  We propose fair mixup, a new data augmentation strategy for imposing the fairness constraint. We show that fairness can be achieved by regularizing the models on paths of interpolated samples between the groups. We empirically show that it ensures a better generalization for both accuracy and fairness measurement in benchmarks.
  arXiv  Detail & Related papers  (2021-03-11T06:57:26Z)
• Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation [3.7819322027528113]
  Measures such as Recall, Precision, F-Measure and Rand Accuracy are biased and should not be used without clear understanding of the biases. We discuss several concepts and measures that reflect the probability that prediction is informed versus chance.
  arXiv  Detail & Related papers  (2020-10-11T02:15:11Z)
• Beyond Individual and Group Fairness [90.4666341812857]
  We present a new data-driven model of fairness that is guided by the unfairness complaints received by the system. Our model supports multiple fairness criteria and takes into account their potential incompatibilities.
  arXiv  Detail & Related papers  (2020-08-21T14:14:44Z)
• On the Relation between Quality-Diversity Evaluation and Distribution-Fitting Goal in Text Generation [86.11292297348622]
  We show that a linear combination of quality and diversity constitutes a divergence metric between the generated distribution and the real distribution. We propose CR/NRR as a substitute for quality/diversity metric pair.
  arXiv  Detail & Related papers  (2020-07-03T04:06:59Z)
• GenDICE: Generalized Offline Estimation of Stationary Values [108.17309783125398]
  We show that effective estimation can still be achieved in important applications. Our approach is based on estimating a ratio that corrects for the discrepancy between the stationary and empirical distributions. The resulting algorithm, GenDICE, is straightforward and effective.
  arXiv  Detail & Related papers  (2020-02-21T00:27:52Z)
• Fast Fair Regression via Efficient Approximations of Mutual Information [0.0]
  This paper introduces fast approximations of the independence, separation and sufficiency group fairness criteria for regression models. It uses such approximations as regularisers to enforce fairness within a regularised risk minimisation framework. Experiments in real-world datasets indicate that in spite of its superior computational efficiency our algorithm still displays state-of-the-art accuracy/fairness tradeoffs.
  arXiv  Detail & Related papers  (2020-02-14T08:50:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.